2024-03-29T11:07:52.116 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:07:52.116 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689 --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689/orig.config.yaml 2024-03-29T11:07:52.151 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:07:52.228 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622689' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpM4AhB3Xc7/K7D8W6puQ5pHgaOxyi5wEP+yhHj9ifzq6krw/XMR2L09TlXLiV16kErqxR0s9oT5hpN8e/DsHQ= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLKHThZ4r0Bb1ok8Y8VwtGYVX4plSSApvkvHIR3vWr7lFP6SSIFvSZm56wznkB3V50jsyn0stxgaz0jPKH27bzk= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:07:52.228 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:07:52.229 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:07:52.229 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:07:52.230 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:07:52.256 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:07:52.256 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:07:52.257 INFO:teuthology.packaging:ref: None 2024-03-29T11:07:52.257 INFO:teuthology.packaging:tag: None 2024-03-29T11:07:52.257 INFO:teuthology.packaging:branch: main 2024-03-29T11:07:52.257 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:07:52.257 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:07:52.625 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:07:52.626 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:07:52.632 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:07:52.632 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:07:52.677 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:07:52.690 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:07:52.701 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:07:52.725 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:07.175750', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpM4AhB3Xc7/K7D8W6puQ5pHgaOxyi5wEP+yhHj9ifzq6krw/XMR2L09TlXLiV16kErqxR0s9oT5hpN8e/DsHQ='} 2024-03-29T11:07:52.745 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:07.177226', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLKHThZ4r0Bb1ok8Y8VwtGYVX4plSSApvkvHIR3vWr7lFP6SSIFvSZm56wznkB3V50jsyn0stxgaz0jPKH27bzk='} 2024-03-29T11:07:52.745 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:07:52.751 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:07:52.751 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:07:52.752 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:07:52.841 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcba8bf60d0>, signals=[15]) 2024-03-29T11:07:52.842 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:07:52.848 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:07:52.848 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-03-29T11:07:52.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:52.955 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-03-29T11:07:52.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:53.045 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:07:53.051 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-03-29T11:07:53.068 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-03-29T11:07:53.069 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-03-29T11:07:53.125 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2024-03-29T11:07:53.125 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="9" 2024-03-29T11:07:53.125 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2024-03-29T11:07:53.125 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:07:53.125 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="9" 2024-03-29T11:07:53.125 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:07:53.125 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:07:53.125 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2024-03-29T11:07:53.125 INFO:teuthology.orchestra.run.smithi059.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:07:53.126 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:07:53.126 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:07:53.126 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:07:53.126 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:07:53.126 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:07:53.126 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-03-29T11:07:53.154 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-03-29T11:07:53.171 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-03-29T11:07:53.171 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-03-29T11:07:53.229 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2024-03-29T11:07:53.229 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="9" 2024-03-29T11:07:53.229 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2024-03-29T11:07:53.229 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:07:53.229 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="9" 2024-03-29T11:07:53.229 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:07:53.229 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:07:53.229 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2024-03-29T11:07:53.229 INFO:teuthology.orchestra.run.smithi172.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:07:53.229 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:07:53.230 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:07:53.230 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:07:53.230 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:07:53.230 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:07:53.230 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-03-29T11:07:53.254 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:07:53.263 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:07:53.285 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:07:53.285 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:07:53.288 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:07:53.304 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:07:53.309 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:07:53.309 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:07:53.346 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:07:53.373 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:07:53.430 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:07:53.466 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:07:53.466 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:07:53.466 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:07:53.467 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:07:53.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:53.483 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-03-29T11:07:53.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:53.484 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-29T11:07:53.542 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:07:53.542 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.14.0-361.el9.x86_64 2024-03-29T11:07:53.542 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2024-03-29T11:07:53.543 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:07:53.543 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.14.0-361.el9.x86_64 2024-03-29T11:07:53.544 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2024-03-29T11:07:55.596 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T11:07:55.666 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T11:07:57.053 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-29T11:07:57.283 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-29T11:07:58.049 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-29T11:08:00.843 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-29T11:08:01.842 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 19 MB 00:02 2024-03-29T11:08:05.734 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.1 MB 00:00 2024-03-29T11:08:07.201 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-29T11:08:08.352 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-29T11:08:13.235 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-29T11:08:14.410 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 19 MB 00:12 2024-03-29T11:08:15.417 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:15.417 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:15.500 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:08:15.509 INFO:teuthology.orchestra.run.smithi172.stdout:Install 4 Packages 2024-03-29T11:08:15.510 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:08:15.510 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 76 M 2024-03-29T11:08:15.510 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 123 M 2024-03-29T11:08:15.510 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:08:15.662 INFO:teuthology.orchestra.run.smithi172.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:08:16.312 INFO:teuthology.orchestra.run.smithi172.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-03-29T11:08:16.637 INFO:teuthology.orchestra.run.smithi172.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 30 MB/s | 34 MB 00:01 2024-03-29T11:08:17.271 INFO:teuthology.orchestra.run.smithi172.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T11:08:17.273 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi172.stdout:Total 43 MB/s | 76 MB 00:01 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:08:17.633 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:08:17.633 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:08:18.491 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:08:18.491 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:08:19.569 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-29T11:08:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:08:21.370 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:21.463 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:21.944 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-29T11:08:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:29.103 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 3.0 MB/s | 21 MB 00:07 2024-03-29T11:08:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:33.992 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:34.262 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T11:08:36.685 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:36.686 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:36.783 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:08:36.788 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:08:36.788 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-29T11:08:36.788 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:08:36.788 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:08:36.788 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:36.788 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-29T11:08:36.788 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:36.788 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:36.788 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:36.788 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:08:36.789 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:08:36.789 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:08:36.789 INFO:teuthology.orchestra.run.smithi059.stdout:Install 4 Packages 2024-03-29T11:08:36.789 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:08:36.789 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 76 M 2024-03-29T11:08:36.789 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 123 M 2024-03-29T11:08:36.790 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:08:36.938 INFO:teuthology.orchestra.run.smithi059.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T11:08:37.565 INFO:teuthology.orchestra.run.smithi059.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-03-29T11:08:37.972 INFO:teuthology.orchestra.run.smithi059.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T11:08:38.314 INFO:teuthology.orchestra.run.smithi059.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T11:08:38.317 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:38.317 INFO:teuthology.orchestra.run.smithi059.stdout:Total 50 MB/s | 76 MB 00:01 2024-03-29T11:08:38.317 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:08:38.738 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:08:38.739 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:08:39.606 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:08:39.607 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:08:42.033 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:08:42.660 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:42.756 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:45.639 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:45.775 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:50.857 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:56.027 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:03.582 INFO:teuthology.orchestra.run.smithi172.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:09:03.625 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:04.467 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:04.467 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:09:04.467 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:09:04.467 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:04.467 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:04.467 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:04.467 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:04.467 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:09:04.467 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:09:04.808 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:04.907 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:04.908 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:04.908 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:04.908 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:04.908 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:04.908 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:09:04.908 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:04.908 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:04.988 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:04.989 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:04.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo rpm -qi grub2-tools 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:Name : grub2-tools 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:Epoch : 1 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:Version : 2.06 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:Release : 68.el9 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: x86_64 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:Group : Unspecified 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:Size : 8264254 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:License : GPLv3+ 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:05.034 INFO:teuthology.orchestra.run.smithi172.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout:Packager : builder@centos.org 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout:Vendor : CentOS 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout:Description : 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout:hardware devices. 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:09:05.035 INFO:teuthology.orchestra.run.smithi172.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:05.036 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:05.036 INFO:teuthology.task.kernel:Updating grub on smithi172 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:05.037 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-03-29T11:09:06.933 INFO:teuthology.orchestra.run.smithi172.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:06.971 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-03-29T11:09:06.973 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-03-29T11:09:06.987 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.wLBmdoDcNY 2024-03-29T11:09:06.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wLBmdoDcNY 2024-03-29T11:09:07.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.wLBmdoDcNY 2024-03-29T11:09:07.190 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.wLBmdoDcNY is 6KB 2024-03-29T11:09:07.210 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.wLBmdoDcNY 2024-03-29T11:09:07.224 DEBUG:teuthology.orchestra.run.smithi172:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:07.293 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:07.294 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:07.294 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:07.294 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:07.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:07.440 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-03-29T11:09:26.874 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:26.874 INFO:teuthology.orchestra.run.smithi059.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:26.874 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:09:26.919 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:27.258 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:27.258 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:27.258 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:27.258 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:27.638 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:27.638 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:09:27.638 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:09:27.638 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:27.638 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:27.638 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:27.638 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:27.639 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:09:27.639 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:09:27.997 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:28.096 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:28.096 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:28.096 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:28.096 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:28.096 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:28.097 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:09:28.097 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:28.097 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:28.179 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:28.180 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:28.180 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2024-03-29T11:09:28.221 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2024-03-29T11:09:28.221 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2024-03-29T11:09:28.221 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.06 2024-03-29T11:09:28.221 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 68.el9 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:Group : Unspecified 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 8264254 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : builder@centos.org 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:28.223 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2024-03-29T11:09:28.223 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:09:28.223 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:28.223 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:28.223 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:28.223 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2024-03-29T11:09:28.223 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:09:28.223 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:28.224 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:28.224 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:28.224 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:28.737 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-03-29T11:09:30.087 INFO:teuthology.orchestra.run.smithi059.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:30.116 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-03-29T11:09:30.119 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-03-29T11:09:30.134 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.81VJj6xENn 2024-03-29T11:09:30.134 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.81VJj6xENn 2024-03-29T11:09:30.205 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.81VJj6xENn 2024-03-29T11:09:30.357 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.81VJj6xENn is 6KB 2024-03-29T11:09:30.405 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.81VJj6xENn 2024-03-29T11:09:30.419 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:30.495 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:30.495 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:30.495 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:30.496 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:30.497 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:30.665 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-03-29T11:09:37.446 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:09:37.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-29T11:09:37.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:55.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-29T11:10:00.670 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:10:00.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-29T11:10:00.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:04.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-29T11:10:04.877 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:07.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-29T11:10:19.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2024-03-29T11:10:19.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-29T11:10:19.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:28.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-29T11:10:28.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:19.991 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:11:28.180 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:11:34.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-29T11:11:35.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:35.339 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-29T11:11:35.745 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-29T11:11:35.746 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:11:35.746 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-29T11:11:35.762 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:11:35.762 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:11:35.763 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:11:35.763 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:11:36.764 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:11:36.764 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:11:36.847 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-03-29T11:11:36.884 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:40.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-29T11:11:40.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:40.528 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-03-29T11:11:40.878 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-29T11:11:40.878 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:11:40.878 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-03-29T11:11:40.893 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:11:40.894 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:11:40.894 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:11:40.894 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:11:41.896 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:11:41.896 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:11:41.957 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2024-03-29T11:11:41.991 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:41.991 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:11:41.999 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:11:42.000 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:11:42.002 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:11:42.021 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:11:42.039 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:11:42.069 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:11:42.069 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:42.073 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:42.163 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:11:42.171 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:11:42.171 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:42.174 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:42.213 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:42.218 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:42.226 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:42.232 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:42.233 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:11:42.242 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:11:42.242 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:42.245 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:42.298 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:11:42.309 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:11:42.309 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:42.316 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:42.354 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:42.416 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:42.463 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:42.548 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:42.596 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:11:42.596 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:42.665 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:42.710 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:42.766 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:42.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:42.898 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:11:42.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:42.968 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-03-29T11:11:42.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-03-29T11:11:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:43.075 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:43.479 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:11:43.486 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:11:43.487 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:11:43.532 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:11:43.568 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2024-03-29T11:11:43.618 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2024-03-29T11:11:43.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2024-03-29T11:11:43.673 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2024-03-29T11:11:43.676 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:11:43.676 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2024-03-29T11:11:43.699 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2024-03-29T11:11:43.699 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2024-03-29T11:11:43.722 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2024-03-29T11:11:43.722 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:11:43.722 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:43.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:43.800 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:11:43.800 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:43.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:43.874 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:11:43.884 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:11:43.904 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:11:43.905 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:16:04.946 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-03-29T11:16:04.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-29T11:16:04.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:05.077 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-03-29T11:16:05.152 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-29T11:16:05.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-29T11:16:05.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:05.239 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-29T11:16:05.320 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-29T11:16:05.321 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:16:05.332 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:16:05.332 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:16:05.332 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:16:05.334 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:16:05.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:16:05.363 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:16:05.380 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:16:05.390 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:16:05.407 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:16:05.410 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2024-03-29T11:16:05.425 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2024-03-29T11:16:05.436 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2024-03-29T11:16:05.444 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:16:05.450 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2024-03-29T11:16:05.461 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:16:05.467 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:16:05.483 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:16:05.519 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:16:05.522 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:16:05.523 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-03-29T11:16:05.523 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:05.523 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:05.523 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:05.523 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:05.534 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:16:05.538 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:16:05.539 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-03-29T11:16:05.539 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:05.539 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:05.539 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:05.539 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:05.539 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:16:05.549 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:16:05.549 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:16:05.549 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:16:05.549 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:16:05.556 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:16:05.556 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:16:05.556 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:16:05.557 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:16:05.559 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:16:05.559 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:16:05.698 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:16:05.698 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:16:05.704 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:16:05.704 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:16:05.869 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:16:05.869 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:16:05.869 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:16:05.898 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:16:05.898 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:16:05.950 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:16:05.951 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:16:05.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:16:05.966 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:16:05.981 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:16:05.981 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:16:06.045 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:16:06.050 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:16:06.112 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2024-03-29T11:16:06.113 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2024-03-29T11:16:06.131 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:16:06.198 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2024-03-29T11:16:06.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2024-03-29T11:16:06.363 INFO:teuthology.orchestra.run.smithi059.stdout:74 files removed 2024-03-29T11:16:06.391 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw 2024-03-29T11:16:06.436 INFO:teuthology.orchestra.run.smithi172.stdout:74 files removed 2024-03-29T11:16:06.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw 2024-03-29T11:16:07.344 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 141 kB/s | 82 kB 00:00 2024-03-29T11:16:07.378 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 120 kB/s | 82 kB 00:00 2024-03-29T11:16:07.686 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 34 kB/s | 10 kB 00:00 2024-03-29T11:16:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-29T11:16:08.120 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-29T11:16:08.346 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-03-29T11:16:08.639 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:16:08.748 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:16:10.031 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-29T11:16:10.199 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-29T11:16:10.905 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-29T11:16:12.714 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.0 MB 00:02 2024-03-29T11:16:15.741 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 19 MB 00:03 2024-03-29T11:16:18.632 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 19 MB 00:04 2024-03-29T11:16:20.340 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.1 MB 00:01 2024-03-29T11:16:22.175 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-29T11:16:22.826 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-29T11:16:24.333 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-29T11:16:25.135 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 31 MB/s | 21 MB 00:00 2024-03-29T11:16:25.142 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 21 MB 00:02 2024-03-29T11:16:30.159 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T11:16:30.469 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T11:16:32.789 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:32.796 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:32.797 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:32.798 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:32.799 INFO:teuthology.orchestra.run.smithi172.stdout:Installing weak dependencies: 2024-03-29T11:16:32.799 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:32.799 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:16:32.799 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:16:32.799 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:16:32.799 INFO:teuthology.orchestra.run.smithi172.stdout:Install 30 Packages 2024-03-29T11:16:32.799 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 2 Packages 2024-03-29T11:16:32.799 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:16:32.800 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 58 M 2024-03-29T11:16:32.800 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:16:33.198 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:16:33.201 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:16:33.201 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:33.202 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse 2024-03-29T11:16:33.203 INFO:teuthology.orchestra.run.smithi059.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:16:33.204 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:16:33.205 INFO:teuthology.orchestra.run.smithi059.stdout:Install 30 Packages 2024-03-29T11:16:33.205 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 2 Packages 2024-03-29T11:16:33.205 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:16:33.205 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 58 M 2024-03-29T11:16:33.205 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:16:40.353 INFO:teuthology.orchestra.run.smithi059.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 782 kB/s | 5.3 MB 00:06 2024-03-29T11:16:40.452 INFO:teuthology.orchestra.run.smithi059.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-29T11:16:41.652 INFO:teuthology.orchestra.run.smithi059.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 618 kB/s | 742 kB 00:01 2024-03-29T11:16:42.569 INFO:teuthology.orchestra.run.smithi059.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 544 kB/s | 498 kB 00:00 2024-03-29T11:16:43.325 INFO:teuthology.orchestra.run.smithi172.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 530 kB/s | 5.3 MB 00:10 2024-03-29T11:16:43.425 INFO:teuthology.orchestra.run.smithi172.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-29T11:16:44.792 INFO:teuthology.orchestra.run.smithi172.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 543 kB/s | 742 kB 00:01 2024-03-29T11:16:45.284 INFO:teuthology.orchestra.run.smithi172.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 811 kB/s | 9.7 MB 00:12 2024-03-29T11:16:45.709 INFO:teuthology.orchestra.run.smithi172.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 544 kB/s | 498 kB 00:00 2024-03-29T11:16:45.884 INFO:teuthology.orchestra.run.smithi172.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 258 kB/s | 45 kB 00:00 2024-03-29T11:16:46.218 INFO:teuthology.orchestra.run.smithi172.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 394 kB/s | 131 kB 00:00 2024-03-29T11:16:46.552 INFO:teuthology.orchestra.run.smithi172.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 486 kB/s | 162 kB 00:00 2024-03-29T11:16:46.553 INFO:teuthology.orchestra.run.smithi059.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 756 kB/s | 9.7 MB 00:13 2024-03-29T11:16:46.728 INFO:teuthology.orchestra.run.smithi059.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 258 kB/s | 45 kB 00:00 2024-03-29T11:16:47.070 INFO:teuthology.orchestra.run.smithi059.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 384 kB/s | 131 kB 00:00 2024-03-29T11:16:47.210 INFO:teuthology.orchestra.run.smithi172.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 487 kB/s | 320 kB 00:00 2024-03-29T11:16:47.395 INFO:teuthology.orchestra.run.smithi059.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 498 kB/s | 162 kB 00:00 2024-03-29T11:16:47.769 INFO:teuthology.orchestra.run.smithi172.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 533 kB/s | 298 kB 00:00 2024-03-29T11:16:47.979 INFO:teuthology.orchestra.run.smithi059.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 549 kB/s | 320 kB 00:00 2024-03-29T11:16:48.010 INFO:teuthology.orchestra.run.smithi172.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 411 kB/s | 99 kB 00:00 2024-03-29T11:16:48.161 INFO:teuthology.orchestra.run.smithi172.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 706 kB/s | 106 kB 00:00 2024-03-29T11:16:48.203 INFO:teuthology.orchestra.run.smithi172.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-29T11:16:48.236 INFO:teuthology.orchestra.run.smithi172.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:16:48.287 INFO:teuthology.orchestra.run.smithi172.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-29T11:16:48.329 INFO:teuthology.orchestra.run.smithi172.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T11:16:48.362 INFO:teuthology.orchestra.run.smithi172.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T11:16:48.396 INFO:teuthology.orchestra.run.smithi172.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.4 MB/s | 220 kB 00:00 2024-03-29T11:16:48.421 INFO:teuthology.orchestra.run.smithi172.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.4 MB/s | 35 kB 00:00 2024-03-29T11:16:48.446 INFO:teuthology.orchestra.run.smithi172.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-29T11:16:48.454 INFO:teuthology.orchestra.run.smithi059.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 627 kB/s | 298 kB 00:00 2024-03-29T11:16:48.629 INFO:teuthology.orchestra.run.smithi059.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 568 kB/s | 99 kB 00:00 2024-03-29T11:16:48.746 INFO:teuthology.orchestra.run.smithi059.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 908 kB/s | 106 kB 00:00 2024-03-29T11:16:48.788 INFO:teuthology.orchestra.run.smithi059.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-29T11:16:48.821 INFO:teuthology.orchestra.run.smithi059.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:16:48.872 INFO:teuthology.orchestra.run.smithi172.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 727 kB/s | 309 kB 00:00 2024-03-29T11:16:48.880 INFO:teuthology.orchestra.run.smithi059.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T11:16:48.922 INFO:teuthology.orchestra.run.smithi059.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T11:16:48.956 INFO:teuthology.orchestra.run.smithi059.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T11:16:48.997 INFO:teuthology.orchestra.run.smithi059.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2024-03-29T11:16:49.031 INFO:teuthology.orchestra.run.smithi059.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-29T11:16:49.065 INFO:teuthology.orchestra.run.smithi059.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T11:16:49.189 INFO:teuthology.orchestra.run.smithi172.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-29T11:16:49.256 INFO:teuthology.orchestra.run.smithi172.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-03-29T11:16:49.315 INFO:teuthology.orchestra.run.smithi059.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-29T11:16:49.322 INFO:teuthology.orchestra.run.smithi172.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 729 kB/s | 49 kB 00:00 2024-03-29T11:16:49.398 INFO:teuthology.orchestra.run.smithi172.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 896 kB/s | 67 kB 00:00 2024-03-29T11:16:49.482 INFO:teuthology.orchestra.run.smithi172.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-29T11:16:49.532 INFO:teuthology.orchestra.run.smithi059.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-03-29T11:16:49.557 INFO:teuthology.orchestra.run.smithi172.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-29T11:16:49.591 INFO:teuthology.orchestra.run.smithi059.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-03-29T11:16:49.655 INFO:teuthology.orchestra.run.smithi172.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-29T11:16:49.655 INFO:teuthology.orchestra.run.smithi059.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 831 kB/s | 49 kB 00:00 2024-03-29T11:16:49.708 INFO:teuthology.orchestra.run.smithi059.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T11:16:49.783 INFO:teuthology.orchestra.run.smithi059.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-29T11:16:49.842 INFO:teuthology.orchestra.run.smithi059.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-29T11:16:49.926 INFO:teuthology.orchestra.run.smithi059.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-29T11:16:51.634 INFO:teuthology.orchestra.run.smithi059.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 576 kB/s | 5.1 MB 00:09 2024-03-29T11:16:53.776 INFO:teuthology.orchestra.run.smithi059.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 897 kB/s | 3.4 MB 00:03 2024-03-29T11:16:53.882 INFO:teuthology.orchestra.run.smithi172.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 607 kB/s | 5.1 MB 00:08 2024-03-29T11:16:54.059 INFO:teuthology.orchestra.run.smithi059.stdout:(31/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 983 kB/s | 20 MB 00:20 2024-03-29T11:16:54.390 INFO:teuthology.orchestra.run.smithi172.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 729 kB/s | 3.4 MB 00:04 2024-03-29T11:16:56.640 INFO:teuthology.orchestra.run.smithi172.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 3.1 MB 00:02 2024-03-29T11:16:56.992 INFO:teuthology.orchestra.run.smithi059.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 598 kB/s | 3.1 MB 00:05 2024-03-29T11:16:56.996 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:56.996 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.4 MB/s | 58 MB 00:23 2024-03-29T11:16:56.996 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:16:57.039 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:16:57.039 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:16:57.530 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:16:57.530 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:16:59.134 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:16:59.225 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:16:59.282 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:16:59.315 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:16:59.337 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:59.353 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:59.512 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:59.568 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:59.913 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:16:59.974 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:00.050 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:00.140 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:17:00.167 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:00.230 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:00.259 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:17:00.295 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:17:00.356 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:00.378 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:00.441 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:00.476 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:17:00.587 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:17:00.628 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:17:00.660 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:17:00.692 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:17:00.721 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:17:00.731 INFO:teuthology.orchestra.run.smithi172.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 734 kB/s | 20 MB 00:27 2024-03-29T11:17:00.736 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:00.736 INFO:teuthology.orchestra.run.smithi172.stdout:Total 2.1 MB/s | 58 MB 00:27 2024-03-29T11:17:00.736 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:17:00.754 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:17:00.774 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:17:00.774 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:17:01.079 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:17:01.109 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:01.190 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:01.222 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:17:01.265 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:17:01.338 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:17:01.338 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:17:01.746 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:17:01.770 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:01.841 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:01.902 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:17:01.930 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:17:02.909 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:17:02.985 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:17:03.019 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:17:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:17:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:03.260 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:03.315 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:17:03.697 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:03.789 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:03.875 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:17:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:03.961 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:03.997 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:17:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:17:04.109 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:04.133 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:17:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:04.340 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:17:04.377 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:04.383 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:17:04.424 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:17:04.456 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:17:04.485 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:17:04.518 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:17:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:17:04.864 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:04.994 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:17:05.026 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:05.034 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:17:05.063 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:05.161 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:05.161 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:17:05.161 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:17:05.161 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:05.189 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:17:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:05.681 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:17:05.711 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:17:07.990 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:08.032 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:08.593 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:08.627 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:08.720 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:08.720 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:17:08.720 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:17:08.720 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:08.762 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:14.951 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:14.951 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2024-03-29T11:17:14.951 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2024-03-29T11:17:14.951 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2024-03-29T11:17:14.952 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2024-03-29T11:17:14.952 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2024-03-29T11:17:14.952 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /root 2024-03-29T11:17:14.952 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2024-03-29T11:17:14.952 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:15.883 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:15.907 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:15.907 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:15.907 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:17:15.907 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:15.907 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:15.907 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:15.909 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:15.994 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:15.994 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:17.119 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:17.119 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:17:17.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:17:17.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:17:17.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:17:17.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:17:17.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:17:17.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:17:17.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:17:17.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:17:17.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:17:17.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:17:17.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:17:17.122 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:17:17.122 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:17:17.122 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:17:17.122 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:17:17.123 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:17:17.123 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:17.123 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:17:17.123 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2024-03-29T11:17:17.967 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2024-03-29T11:17:17.967 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /root 2024-03-29T11:17:17.967 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2024-03-29T11:17:17.967 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:18.120 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:18.120 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:18.120 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:17:18.121 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:18.123 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:17:18.123 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:17:18.123 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:17:18.123 INFO:teuthology.orchestra.run.smithi059.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:17:18.123 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:17:18.123 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:18.123 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:17:18.457 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test 2024-03-29T11:17:18.899 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:18.924 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:18.924 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:18.924 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:17:18.924 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:18.924 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:18.924 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:18.926 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:19.011 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:19.011 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:19.025 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:49 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:17:19.181 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:17:19.181 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:17:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-29T11:17:19.181 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:17:19.181 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-03-29T11:17:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:19.183 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 48 M 2024-03-29T11:17:19.183 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 204 M 2024-03-29T11:17:19.183 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:17:19.648 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-29T11:17:19.707 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-29T11:17:19.749 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-29T11:17:19.999 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 749 kB/s | 187 kB 00:00 2024-03-29T11:17:20.066 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 267 kB/s | 163 kB 00:00 2024-03-29T11:17:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:17:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:17:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:17:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:17:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:17:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:17:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:17:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:17:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:17:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:17:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:17:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:17:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:17:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:17:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:17:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:17:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:17:20.182 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:20.182 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:17:20.182 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:17:21.238 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:21.238 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:21.238 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2024-03-29T11:17:21.238 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.239 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:17:21.241 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.241 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.241 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:21.241 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:17:21.241 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:17:21.241 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:17:21.241 INFO:teuthology.orchestra.run.smithi172.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:17:21.241 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:17:21.241 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:21.241 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:17:21.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-test 2024-03-29T11:17:22.162 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:52 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:17:22.317 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:17:22.317 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:17:22.317 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-29T11:17:22.317 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:17:22.317 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:17:22.317 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:17:22.318 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-29T11:17:22.318 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:17:22.318 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:17:22.318 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:17:22.318 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:17:22.318 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:17:22.318 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:22.318 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:17:22.318 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:17:22.318 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2024-03-29T11:17:22.318 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:22.319 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 48 M 2024-03-29T11:17:22.319 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 204 M 2024-03-29T11:17:22.319 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:17:22.592 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T11:17:22.667 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-29T11:17:22.709 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-29T11:17:22.734 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 547 kB/s | 163 kB 00:00 2024-03-29T11:17:23.760 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 178 kB/s | 187 kB 00:01 2024-03-29T11:17:26.182 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 7.0 MB/s | 47 MB 00:06 2024-03-29T11:17:26.187 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:26.187 INFO:teuthology.orchestra.run.smithi059.stdout:Total 6.9 MB/s | 48 MB 00:07 2024-03-29T11:17:26.203 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:17:26.224 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:17:26.224 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:17:26.619 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:17:26.619 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:17:27.225 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:17:27.292 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:27.330 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:27.370 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:27.415 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:27.446 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:30.109 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 6.2 MB/s | 47 MB 00:07 2024-03-29T11:17:30.114 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:30.114 INFO:teuthology.orchestra.run.smithi172.stdout:Total 6.2 MB/s | 48 MB 00:07 2024-03-29T11:17:30.126 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:17:30.146 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:17:30.146 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:17:30.528 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:17:30.528 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:17:31.104 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:17:31.168 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:31.215 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:31.255 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:31.304 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:31.329 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:32.178 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:32.263 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:34.496 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:34.496 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:17:34.496 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:17:34.496 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:34.496 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:17:34.496 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:17:34.868 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:17:34.868 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:34.868 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:17:34.869 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:34.869 INFO:teuthology.orchestra.run.smithi059.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:17:34.869 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:34.869 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:17:34.869 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:17:34.869 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:17:34.869 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:34.869 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:17:35.155 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph 2024-03-29T11:17:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:17:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:35.892 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:17:35.894 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout:========================================================================================= 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout:========================================================================================= 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:17:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:17:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:17:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:17:35.898 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:35.898 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:17:35.898 INFO:teuthology.orchestra.run.smithi059.stdout:========================================================================================= 2024-03-29T11:17:35.898 INFO:teuthology.orchestra.run.smithi059.stdout:Install 36 Packages 2024-03-29T11:17:35.898 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:35.898 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 30 M 2024-03-29T11:17:35.898 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 107 M 2024-03-29T11:17:35.899 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:17:36.829 INFO:teuthology.orchestra.run.smithi059.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-29T11:17:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:17:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:17:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:17:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:17:38.571 INFO:teuthology.orchestra.run.smithi059.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 723 kB/s | 1.5 MB 00:02 2024-03-29T11:17:38.691 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:17:38.691 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:38.692 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:17:38.692 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:38.692 INFO:teuthology.orchestra.run.smithi172.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:17:38.692 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:38.692 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:17:38.692 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:17:38.693 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:17:38.693 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:38.693 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:17:38.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph 2024-03-29T11:17:39.088 INFO:teuthology.orchestra.run.smithi059.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 828 kB/s | 2.1 MB 00:02 2024-03-29T11:17:39.338 INFO:teuthology.orchestra.run.smithi059.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 994 kB/s | 247 kB 00:00 2024-03-29T11:17:39.455 INFO:teuthology.orchestra.run.smithi059.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:17:39.497 INFO:teuthology.orchestra.run.smithi059.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:17:39.530 INFO:teuthology.orchestra.run.smithi059.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-29T11:17:39.548 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:17:39.564 INFO:teuthology.orchestra.run.smithi059.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:17:39.597 INFO:teuthology.orchestra.run.smithi059.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:17:39.639 INFO:teuthology.orchestra.run.smithi059.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:17:39.706 INFO:teuthology.orchestra.run.smithi059.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:17:39.709 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:17:39.712 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2024-03-29T11:17:39.712 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-29T11:17:39.712 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2024-03-29T11:17:39.712 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:17:39.712 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:17:39.712 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-29T11:17:39.712 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:17:39.712 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:17:39.712 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:17:39.712 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:17:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:17:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi172.stdout:Install 36 Packages 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:39.716 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 30 M 2024-03-29T11:17:39.716 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 107 M 2024-03-29T11:17:39.716 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:17:39.740 INFO:teuthology.orchestra.run.smithi059.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:17:39.773 INFO:teuthology.orchestra.run.smithi059.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:17:39.807 INFO:teuthology.orchestra.run.smithi059.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:17:39.932 INFO:teuthology.orchestra.run.smithi059.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 998 kB/s | 125 kB 00:00 2024-03-29T11:17:40.008 INFO:teuthology.orchestra.run.smithi059.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 298 kB/s | 22 kB 00:00 2024-03-29T11:17:40.216 INFO:teuthology.orchestra.run.smithi059.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 725 kB/s | 151 kB 00:00 2024-03-29T11:17:40.275 INFO:teuthology.orchestra.run.smithi059.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 749 kB/s | 43 kB 00:00 2024-03-29T11:17:40.367 INFO:teuthology.orchestra.run.smithi059.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-29T11:17:40.467 INFO:teuthology.orchestra.run.smithi059.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-29T11:17:40.520 INFO:teuthology.orchestra.run.smithi172.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-29T11:17:40.525 INFO:teuthology.orchestra.run.smithi059.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-03-29T11:17:40.576 INFO:teuthology.orchestra.run.smithi059.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-03-29T11:17:40.635 INFO:teuthology.orchestra.run.smithi059.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 399 kB/s | 23 kB 00:00 2024-03-29T11:17:40.693 INFO:teuthology.orchestra.run.smithi059.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 333 kB/s | 19 kB 00:00 2024-03-29T11:17:40.752 INFO:teuthology.orchestra.run.smithi059.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2024-03-29T11:17:40.844 INFO:teuthology.orchestra.run.smithi059.stdout:(26/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.2 MB/s | 4.7 MB 00:04 2024-03-29T11:17:40.869 INFO:teuthology.orchestra.run.smithi059.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 395 kB/s | 46 kB 00:00 2024-03-29T11:17:40.936 INFO:teuthology.orchestra.run.smithi059.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-29T11:17:40.995 INFO:teuthology.orchestra.run.smithi059.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-03-29T11:17:41.020 INFO:teuthology.orchestra.run.smithi059.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 448 kB/s | 79 kB 00:00 2024-03-29T11:17:41.054 INFO:teuthology.orchestra.run.smithi059.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-29T11:17:41.087 INFO:teuthology.orchestra.run.smithi059.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 535 kB/s | 36 kB 00:00 2024-03-29T11:17:41.121 INFO:teuthology.orchestra.run.smithi059.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-29T11:17:41.180 INFO:teuthology.orchestra.run.smithi059.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-29T11:17:41.246 INFO:teuthology.orchestra.run.smithi059.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-03-29T11:17:41.812 INFO:teuthology.orchestra.run.smithi172.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 2.1 MB 00:01 2024-03-29T11:17:41.912 INFO:teuthology.orchestra.run.smithi172.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 841 kB/s | 1.5 MB 00:01 2024-03-29T11:17:42.087 INFO:teuthology.orchestra.run.smithi172.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.4 MB/s | 247 kB 00:00 2024-03-29T11:17:42.221 INFO:teuthology.orchestra.run.smithi172.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T11:17:42.263 INFO:teuthology.orchestra.run.smithi172.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:17:42.297 INFO:teuthology.orchestra.run.smithi172.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-03-29T11:17:42.330 INFO:teuthology.orchestra.run.smithi172.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:17:42.364 INFO:teuthology.orchestra.run.smithi172.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:17:42.406 INFO:teuthology.orchestra.run.smithi172.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:17:42.473 INFO:teuthology.orchestra.run.smithi172.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:17:42.506 INFO:teuthology.orchestra.run.smithi172.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:17:42.540 INFO:teuthology.orchestra.run.smithi172.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:17:42.573 INFO:teuthology.orchestra.run.smithi172.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:17:42.766 INFO:teuthology.orchestra.run.smithi172.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 652 kB/s | 125 kB 00:00 2024-03-29T11:17:42.924 INFO:teuthology.orchestra.run.smithi172.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-29T11:17:43.058 INFO:teuthology.orchestra.run.smithi172.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-29T11:17:43.108 INFO:teuthology.orchestra.run.smithi172.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 869 kB/s | 43 kB 00:00 2024-03-29T11:17:43.166 INFO:teuthology.orchestra.run.smithi172.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-03-29T11:17:43.233 INFO:teuthology.orchestra.run.smithi172.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-29T11:17:43.276 INFO:teuthology.orchestra.run.smithi172.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-03-29T11:17:43.317 INFO:teuthology.orchestra.run.smithi172.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 427 kB/s | 18 kB 00:00 2024-03-29T11:17:43.359 INFO:teuthology.orchestra.run.smithi172.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 556 kB/s | 23 kB 00:00 2024-03-29T11:17:43.426 INFO:teuthology.orchestra.run.smithi172.stdout:(24/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.6 MB/s | 4.7 MB 00:02 2024-03-29T11:17:43.452 INFO:teuthology.orchestra.run.smithi172.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-29T11:17:43.494 INFO:teuthology.orchestra.run.smithi172.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-29T11:17:43.519 INFO:teuthology.orchestra.run.smithi172.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2024-03-29T11:17:43.544 INFO:teuthology.orchestra.run.smithi172.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T11:17:43.586 INFO:teuthology.orchestra.run.smithi172.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2024-03-29T11:17:43.620 INFO:teuthology.orchestra.run.smithi172.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.7 MB/s | 90 kB 00:00 2024-03-29T11:17:43.654 INFO:teuthology.orchestra.run.smithi172.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-29T11:17:43.679 INFO:teuthology.orchestra.run.smithi172.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 609 kB/s | 36 kB 00:00 2024-03-29T11:17:43.712 INFO:teuthology.orchestra.run.smithi172.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-29T11:17:43.738 INFO:teuthology.orchestra.run.smithi172.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-03-29T11:17:43.763 INFO:teuthology.orchestra.run.smithi172.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-03-29T11:17:44.755 INFO:teuthology.orchestra.run.smithi172.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 5.7 MB/s | 17 MB 00:02 2024-03-29T11:17:44.758 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:44.758 INFO:teuthology.orchestra.run.smithi172.stdout:Total 5.9 MB/s | 30 MB 00:05 2024-03-29T11:17:44.784 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:17:44.827 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:17:44.827 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:17:45.251 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:17:45.251 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:17:46.570 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:17:46.624 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:46.763 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:46.808 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:46.841 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:47.061 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:47.132 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:47.329 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:47.652 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:47.681 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:47.791 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:47.824 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:47.861 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:47.900 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:47.943 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:48.034 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:48.286 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:48.591 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:48.620 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:48.647 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:48.647 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:48.647 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:48.647 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:49.838 INFO:teuthology.orchestra.run.smithi059.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.5 MB/s | 17 MB 00:11 2024-03-29T11:17:49.842 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:49.842 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.1 MB/s | 30 MB 00:13 2024-03-29T11:17:49.867 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:17:49.912 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:17:49.912 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:17:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:50.114 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:50.114 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:50.114 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:50.114 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:50.114 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:50.114 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:50.324 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:17:50.324 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:17:50.555 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:50.579 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:50.579 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:50.579 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:50.579 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:50.579 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:50.579 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:50.812 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:50.812 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:50.812 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:50.812 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:17:51.833 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:51.872 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:51.901 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:51.974 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:52.017 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:52.049 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:52.144 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:52.221 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:52.404 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:52.441 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:52.500 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:52.562 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:52.677 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:52.677 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:52.677 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:52.677 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:52.677 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:52.677 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:52.677 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:52.677 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:52.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:52.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:52.755 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:52.803 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:52.907 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:52.986 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:53.017 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:53.050 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:53.091 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:53.142 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:53.176 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:53.270 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:53.308 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:53.382 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:53.525 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:53.596 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:53.596 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:53.596 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:17:53.596 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:53.596 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:53.596 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:53.596 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:53.596 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:53.597 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:53.598 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:53.599 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:53.599 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:53.599 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:53.599 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:17:53.654 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:53.848 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:53.878 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:53.903 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:53.903 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:53.903 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:53.903 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:53.903 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:53.903 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:53.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-base 2024-03-29T11:17:54.526 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:24 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:17:54.627 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:54.680 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:17:54.681 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-29T11:17:54.681 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:17:54.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install cephadm 2024-03-29T11:17:55.324 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:17:55.421 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:55.447 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:55.447 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:55.447 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:55.447 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:55.448 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:55.448 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:55.477 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:17:55.478 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:17:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-29T11:17:55.478 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:17:55.478 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:17:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:55.478 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:17:55.478 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:17:55.478 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-29T11:17:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:55.479 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 747 k 2024-03-29T11:17:55.479 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 754 k 2024-03-29T11:17:55.479 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:17:55.886 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.8 MB/s | 747 kB 00:00 2024-03-29T11:17:55.887 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:55.887 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.8 MB/s | 747 kB 00:00 2024-03-29T11:17:55.887 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:17:55.893 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:17:55.893 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:17:55.898 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:17:55.899 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:17:55.905 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:55.928 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:55.928 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:55.929 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:55.929 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:55.929 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:55.929 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:17:56.147 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:56.172 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:56.172 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:56.172 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:56.173 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:56.173 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:56.173 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:56.239 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:56.386 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:56.475 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:56.867 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:57.222 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:57.222 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:57.222 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:17:57.222 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:57.222 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:57.222 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:17:57.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:58.001 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:17:58.069 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:58.069 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:58.069 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:58.069 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:58.070 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:58.071 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:58.071 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:58.071 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:58.071 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:58.071 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:58.071 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:58.071 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:58.071 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:58.072 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:58.072 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:58.072 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:58.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:58.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:58.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:58.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:58.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:58.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:58.073 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:58.158 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:17:58.159 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:17:58.159 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-03-29T11:17:58.159 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:17:58.159 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:17:58.160 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:58.160 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:58.160 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:17:58.160 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:17:58.160 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-29T11:17:58.160 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:58.160 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 145 k 2024-03-29T11:17:58.160 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 436 k 2024-03-29T11:17:58.160 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:17:58.592 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 336 kB/s | 145 kB 00:00 2024-03-29T11:17:58.593 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:58.593 INFO:teuthology.orchestra.run.smithi172.stdout:Total 335 kB/s | 145 kB 00:00 2024-03-29T11:17:58.593 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:17:58.601 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:17:58.601 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:17:58.654 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:17:58.655 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:17:58.795 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:17:58.860 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:58.884 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:58.884 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:58.884 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:58.885 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:58.962 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:58.963 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:58.964 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:17:58.965 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:17:59.273 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base 2024-03-29T11:17:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:59.792 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:17:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:17:59.792 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:17:59.869 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:17:59.973 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:00.029 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:00.030 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-29T11:18:00.030 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:00.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr 2024-03-29T11:18:00.092 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install cephadm 2024-03-29T11:18:00.640 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:00.671 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:00.739 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:00.790 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:00.791 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-29T11:18:00.792 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:00.828 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:00.829 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:00.829 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-29T11:18:00.830 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:00.830 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:00.830 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:18:00.830 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:00.830 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:00.830 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:00.830 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-29T11:18:00.831 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:00.831 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 747 k 2024-03-29T11:18:00.831 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 754 k 2024-03-29T11:18:00.831 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:00.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:18:01.186 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.1 MB/s | 747 kB 00:00 2024-03-29T11:18:01.187 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:01.187 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.0 MB/s | 747 kB 00:00 2024-03-29T11:18:01.187 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:01.192 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:01.192 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:01.198 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:01.199 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:01.277 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:01.439 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:01.592 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:01.593 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-03-29T11:18:01.593 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-29T11:18:01.593 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-03-29T11:18:01.593 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:18:01.593 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:18:01.593 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-29T11:18:01.593 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:18:01.593 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:18:01.593 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout:Install 12 Packages 2024-03-29T11:18:01.594 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:01.595 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 6.9 M 2024-03-29T11:18:01.595 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 26 M 2024-03-29T11:18:01.595 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:18:01.728 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:01.818 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:02.146 INFO:teuthology.orchestra.run.smithi172.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 82 kB/s | 15 kB 00:00 2024-03-29T11:18:02.159 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:02.163 INFO:teuthology.orchestra.run.smithi172.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 154 kB/s | 31 kB 00:00 2024-03-29T11:18:02.380 INFO:teuthology.orchestra.run.smithi172.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 238 kB 00:00 2024-03-29T11:18:02.430 INFO:teuthology.orchestra.run.smithi172.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-03-29T11:18:02.466 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:02.467 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:02.467 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:02.467 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:02.467 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:02.467 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:02.680 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:18:02.706 INFO:teuthology.orchestra.run.smithi172.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.6 MB/s | 863 kB 00:00 2024-03-29T11:18:02.739 INFO:teuthology.orchestra.run.smithi172.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-03-29T11:18:02.815 INFO:teuthology.orchestra.run.smithi172.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.1 MB/s | 1.8 MB 00:00 2024-03-29T11:18:02.840 INFO:teuthology.orchestra.run.smithi172.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 145 kB/s | 19 kB 00:00 2024-03-29T11:18:02.957 INFO:teuthology.orchestra.run.smithi172.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.0 MB/s | 144 kB 00:00 2024-03-29T11:18:02.999 INFO:teuthology.orchestra.run.smithi172.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.9 MB/s | 2.0 MB 00:00 2024-03-29T11:18:03.024 INFO:teuthology.orchestra.run.smithi172.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 167 kB/s | 31 kB 00:00 2024-03-29T11:18:03.049 INFO:teuthology.orchestra.run.smithi172.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-29T11:18:03.052 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:03.052 INFO:teuthology.orchestra.run.smithi172.stdout:Total 4.7 MB/s | 6.9 MB 00:01 2024-03-29T11:18:03.069 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:18:03.086 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:18:03.086 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:18:03.256 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:03.261 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:18:03.262 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:18:03.414 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:03.415 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-03-29T11:18:03.415 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:03.415 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:18:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:03.416 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:03.416 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:03.416 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-29T11:18:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:03.416 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 145 k 2024-03-29T11:18:03.416 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 436 k 2024-03-29T11:18:03.416 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:03.780 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:18:03.782 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 397 kB/s | 145 kB 00:00 2024-03-29T11:18:03.783 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:03.783 INFO:teuthology.orchestra.run.smithi059.stdout:Total 395 kB/s | 145 kB 00:00 2024-03-29T11:18:03.783 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:03.792 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:03.792 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:03.848 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:03.848 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:03.874 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:18:03.910 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:18:03.956 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:18:03.987 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:18:03.994 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:04.050 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:04.074 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:04.074 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:04.074 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:18:04.074 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:18:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:18:04.321 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:18:04.366 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:18:04.394 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:04.426 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:18:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:18:04.726 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:05.216 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:05.216 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:05.216 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:05.217 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:05.217 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:05.217 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:05.453 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr 2024-03-29T11:18:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:18:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:18:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:18:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:18:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:18:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:18:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:18:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:18:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:18:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:18:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:18:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:05.888 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:05.889 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:06.031 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:06.133 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:06.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:18:06.188 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:06.189 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-29T11:18:06.189 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:06.260 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:18:06.722 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:06.830 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:06.887 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:18:06.888 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:18:06.889 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:18:06.889 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:18:06.889 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:18:06.889 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:18:06.889 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:18:06.889 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:06.889 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:18:06.889 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-29T11:18:06.889 INFO:teuthology.orchestra.run.smithi172.stdout:Install 12 Packages 2024-03-29T11:18:06.889 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:06.890 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 37 M 2024-03-29T11:18:06.890 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 209 M 2024-03-29T11:18:06.890 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:18:06.987 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:06.988 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-03-29T11:18:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-29T11:18:06.988 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-03-29T11:18:06.988 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:18:06.988 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-29T11:18:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:18:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:18:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:18:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:18:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:18:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:18:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:18:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:18:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:18:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:18:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:18:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:06.989 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:06.989 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-03-29T11:18:06.989 INFO:teuthology.orchestra.run.smithi059.stdout:Install 12 Packages 2024-03-29T11:18:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:06.990 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 6.9 M 2024-03-29T11:18:06.990 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 26 M 2024-03-29T11:18:06.990 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:07.189 INFO:teuthology.orchestra.run.smithi172.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 222 kB/s | 32 kB 00:00 2024-03-29T11:18:07.223 INFO:teuthology.orchestra.run.smithi172.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 521 kB/s | 17 kB 00:00 2024-03-29T11:18:07.290 INFO:teuthology.orchestra.run.smithi172.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-29T11:18:07.315 INFO:teuthology.orchestra.run.smithi172.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 381 kB/s | 34 kB 00:00 2024-03-29T11:18:07.507 INFO:teuthology.orchestra.run.smithi172.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-29T11:18:07.574 INFO:teuthology.orchestra.run.smithi172.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T11:18:07.599 INFO:teuthology.orchestra.run.smithi172.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-29T11:18:07.628 INFO:teuthology.orchestra.run.smithi059.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 64 kB/s | 15 kB 00:00 2024-03-29T11:18:07.761 INFO:teuthology.orchestra.run.smithi059.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 85 kB/s | 31 kB 00:00 2024-03-29T11:18:07.895 INFO:teuthology.orchestra.run.smithi059.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2024-03-29T11:18:07.929 INFO:teuthology.orchestra.run.smithi059.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 238 kB 00:00 2024-03-29T11:18:07.967 INFO:teuthology.orchestra.run.smithi172.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-29T11:18:08.058 INFO:teuthology.orchestra.run.smithi172.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 7.3 MB/s | 7.4 MB 00:01 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi172.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-29T11:18:08.175 INFO:teuthology.orchestra.run.smithi172.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 939 kB/s | 195 kB 00:00 2024-03-29T11:18:08.204 INFO:teuthology.orchestra.run.smithi059.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-03-29T11:18:08.208 INFO:teuthology.orchestra.run.smithi172.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-29T11:18:08.211 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:08.211 INFO:teuthology.orchestra.run.smithi172.stdout:Total 28 MB/s | 37 MB 00:01 2024-03-29T11:18:08.246 INFO:teuthology.orchestra.run.smithi059.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.1 MB/s | 1.8 MB 00:00 2024-03-29T11:18:08.250 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:18:08.274 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:18:08.274 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:18:08.405 INFO:teuthology.orchestra.run.smithi059.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:18:08.455 INFO:teuthology.orchestra.run.smithi059.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-03-29T11:18:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-03-29T11:18:08.563 INFO:teuthology.orchestra.run.smithi059.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 283 kB/s | 31 kB 00:00 2024-03-29T11:18:08.588 INFO:teuthology.orchestra.run.smithi059.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-29T11:18:08.645 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:18:08.645 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:18:08.814 INFO:teuthology.orchestra.run.smithi059.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.6 MB/s | 2.0 MB 00:00 2024-03-29T11:18:08.819 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:08.819 INFO:teuthology.orchestra.run.smithi059.stdout:Total 3.7 MB/s | 6.9 MB 00:01 2024-03-29T11:18:08.838 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:08.856 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:08.856 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:09.035 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:09.035 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:18:09.423 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:18:09.454 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:18:09.551 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:09.556 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:09.583 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:18:09.631 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:18:09.675 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:18:09.723 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:18:09.764 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:09.776 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:18:09.799 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:09.886 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:18:09.892 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:18:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:18:10.117 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:18:10.165 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:18:10.201 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:10.233 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:18:10.498 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:18:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:10.546 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:18:10.653 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:18:11.239 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:11.239 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:18:11.239 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:18:11.239 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:18:11.239 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:18:11.239 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:18:11.239 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:18:11.239 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:18:11.239 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:18:11.240 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:11.240 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:18:11.240 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:18:11.608 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:18:11.608 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:11.608 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:11.608 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:18:11.608 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:11.609 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:11.875 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:18:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:18:12.445 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:12.604 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:12.605 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-03-29T11:18:12.605 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-29T11:18:12.605 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-03-29T11:18:12.605 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:12.605 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:18:12.605 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:12.606 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-03-29T11:18:12.607 INFO:teuthology.orchestra.run.smithi059.stdout:Install 12 Packages 2024-03-29T11:18:12.607 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:12.607 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 37 M 2024-03-29T11:18:12.607 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 209 M 2024-03-29T11:18:12.607 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:12.963 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:18:12.997 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:13.286 INFO:teuthology.orchestra.run.smithi059.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 240 kB/s | 32 kB 00:00 2024-03-29T11:18:13.320 INFO:teuthology.orchestra.run.smithi059.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 532 kB/s | 17 kB 00:00 2024-03-29T11:18:13.353 INFO:teuthology.orchestra.run.smithi059.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:18:13.429 INFO:teuthology.orchestra.run.smithi059.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T11:18:14.188 INFO:teuthology.orchestra.run.smithi059.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 5.7 MB/s | 4.7 MB 00:00 2024-03-29T11:18:14.246 INFO:teuthology.orchestra.run.smithi059.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.9 MB/s | 4.8 MB 00:00 2024-03-29T11:18:14.271 INFO:teuthology.orchestra.run.smithi059.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-29T11:18:14.613 INFO:teuthology.orchestra.run.smithi059.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-29T11:18:14.638 INFO:teuthology.orchestra.run.smithi059.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-29T11:18:14.714 INFO:teuthology.orchestra.run.smithi059.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-03-29T11:18:14.805 INFO:teuthology.orchestra.run.smithi059.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-29T11:18:14.947 INFO:teuthology.orchestra.run.smithi059.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.1 MB/s | 7.4 MB 00:01 2024-03-29T11:18:14.949 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:14.950 INFO:teuthology.orchestra.run.smithi059.stdout:Total 16 MB/s | 37 MB 00:02 2024-03-29T11:18:14.989 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:15.014 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:15.014 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:15.397 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:15.397 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:15.399 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:15.399 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:18:15.399 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:18:15.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:15.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:18:15.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:15.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:15.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:18:15.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:18:15.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:18:15.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:18:15.400 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:18:15.842 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:18:15.842 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:15.842 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:18:15.842 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:15.842 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:18:15.843 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:18:15.843 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:18:15.843 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:18:15.843 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:18:15.843 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:18:15.843 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:18:15.843 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:18:15.843 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:15.843 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:15.844 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:18:15.844 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:15.844 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:16.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:18:16.266 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:16.337 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:18:16.368 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:18:16.466 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:16.505 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:18:16.689 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:16.720 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:16.802 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:16.806 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:18:16.955 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:16.956 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-29T11:18:16.956 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-29T11:18:16.956 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-29T11:18:16.956 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:18:16.956 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:18:16.956 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-29T11:18:16.956 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:18:16.956 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:18:16.956 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout:Install 13 Packages 2024-03-29T11:18:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:16.958 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 2.0 M 2024-03-29T11:18:16.958 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 26 M 2024-03-29T11:18:16.958 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:18:17.131 INFO:teuthology.orchestra.run.smithi172.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 196 kB/s | 24 kB 00:00 2024-03-29T11:18:17.156 INFO:teuthology.orchestra.run.smithi172.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-03-29T11:18:17.206 INFO:teuthology.orchestra.run.smithi172.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:18:17.231 INFO:teuthology.orchestra.run.smithi172.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:18:17.257 INFO:teuthology.orchestra.run.smithi172.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:18:17.282 INFO:teuthology.orchestra.run.smithi172.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-29T11:18:17.424 INFO:teuthology.orchestra.run.smithi172.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 125 kB/s | 52 kB 00:00 2024-03-29T11:18:17.449 INFO:teuthology.orchestra.run.smithi172.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-03-29T11:18:17.474 INFO:teuthology.orchestra.run.smithi172.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-03-29T11:18:17.492 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:18:17.541 INFO:teuthology.orchestra.run.smithi172.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.7 MB/s | 203 kB 00:00 2024-03-29T11:18:17.566 INFO:teuthology.orchestra.run.smithi172.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 644 kB/s | 59 kB 00:00 2024-03-29T11:18:17.612 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:18:17.617 INFO:teuthology.orchestra.run.smithi172.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-29T11:18:17.659 INFO:teuthology.orchestra.run.smithi172.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.6 MB/s | 1.0 MB 00:00 2024-03-29T11:18:17.663 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:17.663 INFO:teuthology.orchestra.run.smithi172.stdout:Total 2.9 MB/s | 2.0 MB 00:00 2024-03-29T11:18:17.663 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:18:17.679 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:18:17.679 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:18:17.769 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:18:17.769 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:18:18.295 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:18:18.457 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:18:18.518 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:18:18.574 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:18:18.619 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:18:18.660 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:18:18.755 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:18:18.863 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:18:18.920 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:18:18.933 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:18:19.656 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:18:19.700 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:18:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:18:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:18:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:19.999 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:18:20.018 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:20.577 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:20.577 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:18:20.577 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:18:20.577 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:18:20.577 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:18:20.577 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:18:20.577 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:18:20.577 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:18:20.577 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:18:20.578 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:18:20.578 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:18:20.578 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:18:20.578 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:18:20.936 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:18:20.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:18:20.938 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:20.938 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:21.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:21.797 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:21.949 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:21.950 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-29T11:18:21.950 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-29T11:18:21.950 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-29T11:18:21.950 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:18:21.950 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:21.950 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-29T11:18:21.950 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:21.951 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:21.951 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:21.951 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:21.951 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:21.951 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:21.951 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:18:21.951 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-29T11:18:21.951 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2024-03-29T11:18:21.951 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:21.952 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 6.8 M 2024-03-29T11:18:21.952 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 33 M 2024-03-29T11:18:21.952 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:18:22.376 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-29T11:18:22.460 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 737 kB/s | 248 kB 00:00 2024-03-29T11:18:22.504 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:22.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:18:22.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:18:22.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:22.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:18:22.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:22.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:22.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:18:22.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:18:22.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:18:22.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:18:22.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:18:22.518 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 349 kB/s | 138 kB 00:00 2024-03-29T11:18:22.886 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 144 kB/s | 58 kB 00:00 2024-03-29T11:18:22.911 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 130 kB/s | 51 kB 00:00 2024-03-29T11:18:22.933 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:18:22.933 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:22.933 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:22.933 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:22.933 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:18:22.933 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:18:22.934 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:18:22.934 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:18:22.934 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:18:22.934 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:18:22.934 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:18:22.934 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:18:22.934 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:22.934 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:22.934 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:18:22.934 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:22.934 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:23.036 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 832 kB/s | 548 kB 00:00 2024-03-29T11:18:23.040 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:23.040 INFO:teuthology.orchestra.run.smithi172.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2024-03-29T11:18:23.053 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:18:23.064 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:18:23.064 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:18:23.153 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:18:23.154 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:18:23.386 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:18:23.461 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:18:23.635 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:23.696 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:23.960 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:24.119 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:24.120 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-03-29T11:18:24.120 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-29T11:18:24.120 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-03-29T11:18:24.120 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:24.120 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:18:24.120 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-29T11:18:24.120 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:18:24.120 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2024-03-29T11:18:24.121 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:24.122 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.0 M 2024-03-29T11:18:24.122 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 26 M 2024-03-29T11:18:24.122 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:24.207 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:24.281 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:24.300 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 145 kB/s | 18 kB 00:00 2024-03-29T11:18:24.325 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 163 kB/s | 24 kB 00:00 2024-03-29T11:18:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:24.384 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-29T11:18:24.410 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-29T11:18:24.435 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:18:24.460 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-29T11:18:24.602 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 106 kB/s | 15 kB 00:00 2024-03-29T11:18:24.627 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-03-29T11:18:24.661 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 108 kB/s | 52 kB 00:00 2024-03-29T11:18:24.745 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.4 MB/s | 203 kB 00:00 2024-03-29T11:18:24.811 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-29T11:18:24.853 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-03-29T11:18:24.970 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-03-29T11:18:24.972 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:24.972 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-29T11:18:24.972 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:24.987 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:24.987 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:25.084 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:25.084 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:25.239 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:25.239 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:25.239 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:25.239 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:25.239 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:25.239 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:25.553 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:18:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:25.553 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:25.782 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:18:25.813 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-fuse 2024-03-29T11:18:25.838 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:18:25.896 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:18:25.930 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:18:25.972 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:18:26.072 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:18:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:18:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:18:26.373 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:26.525 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:26.525 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:26.525 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-29T11:18:26.525 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:26.525 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:18:26.526 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:26.526 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-29T11:18:26.526 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:26.526 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:26.526 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:18:26.526 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:26.526 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2024-03-29T11:18:26.526 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:26.527 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 964 k 2024-03-29T11:18:26.527 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 2.8 M 2024-03-29T11:18:26.527 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:18:26.664 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 575 kB/s | 79 kB 00:00 2024-03-29T11:18:26.995 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:18:27.045 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:18:27.079 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:18:27.128 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:18:27.156 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:27.539 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 874 kB/s | 885 kB 00:01 2024-03-29T11:18:27.540 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:27.541 INFO:teuthology.orchestra.run.smithi172.stdout:Total 950 kB/s | 964 kB 00:01 2024-03-29T11:18:27.541 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:18:27.562 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:18:27.562 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:18:27.637 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:18:27.637 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:18:27.815 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:18:27.948 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:27.954 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:18:27.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:18:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:28.326 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:18:28.326 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:28.326 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:28.326 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:28.326 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:18:28.326 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:18:28.326 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:18:28.326 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:28.601 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:28.758 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:28.758 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:29.076 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:18:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:29.076 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:29.178 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:29.336 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:29.336 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-03-29T11:18:29.336 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-03-29T11:18:29.337 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-03-29T11:18:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:29.338 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 6.8 M 2024-03-29T11:18:29.338 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 33 M 2024-03-29T11:18:29.338 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:29.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-volume 2024-03-29T11:18:29.563 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-03-29T11:18:29.738 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 452 kB/s | 138 kB 00:00 2024-03-29T11:18:29.938 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:30.088 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.8 MB/s | 5.8 MB 00:00 2024-03-29T11:18:30.091 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:30.092 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-29T11:18:30.092 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:30.092 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:18:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:30.092 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-29T11:18:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:30.093 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:18:30.093 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:30.093 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2024-03-29T11:18:30.093 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:30.093 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 332 k 2024-03-29T11:18:30.093 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 1.5 M 2024-03-29T11:18:30.093 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:18:30.210 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 592 kB/s | 69 kB 00:00 2024-03-29T11:18:30.305 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 102 kB/s | 58 kB 00:00 2024-03-29T11:18:30.538 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 562 kB/s | 548 kB 00:00 2024-03-29T11:18:30.576 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 546 kB/s | 264 kB 00:00 2024-03-29T11:18:30.578 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:30.578 INFO:teuthology.orchestra.run.smithi172.stdout:Total 685 kB/s | 332 kB 00:00 2024-03-29T11:18:30.578 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:18:30.587 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:18:30.587 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:18:30.630 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:18:30.630 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:18:30.655 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 90 kB/s | 51 kB 00:00 2024-03-29T11:18:30.656 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:30.657 INFO:teuthology.orchestra.run.smithi059.stdout:Total 5.2 MB/s | 6.8 MB 00:01 2024-03-29T11:18:30.672 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:30.683 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:30.684 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:30.775 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:18:30.779 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:30.779 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:30.956 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:31.012 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:31.029 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:31.029 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:31.029 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:31.029 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:31.085 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:31.272 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:31.325 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:31.717 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:31.877 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:31.971 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:32.048 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:32.061 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:32.062 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:32.062 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:18:32.062 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:32.062 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:32.062 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:32.062 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:32.077 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:32.424 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados-devel 2024-03-29T11:18:32.830 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:32.830 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:32.830 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:32.830 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:32.830 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:32.830 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:33.010 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:03 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:33.148 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:33.148 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:33.148 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:33.148 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:33.148 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:33.148 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:33.148 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:33.148 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:33.148 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:33.148 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:33.149 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:33.161 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:33.162 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-03-29T11:18:33.162 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:33.162 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:18:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:33.162 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:18:33.162 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:33.162 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-29T11:18:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:33.163 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 127 k 2024-03-29T11:18:33.163 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 456 k 2024-03-29T11:18:33.163 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:18:33.378 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse 2024-03-29T11:18:33.390 INFO:teuthology.orchestra.run.smithi172.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 559 kB/s | 127 kB 00:00 2024-03-29T11:18:33.391 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:33.391 INFO:teuthology.orchestra.run.smithi172.stdout:Total 555 kB/s | 127 kB 00:00 2024-03-29T11:18:33.391 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:18:33.397 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:18:33.397 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:18:33.452 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:18:33.452 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:18:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:18:33.698 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:03 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:34.108 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:34.108 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:34.108 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-29T11:18:34.108 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:34.108 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:34.109 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:34.109 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-29T11:18:34.109 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:34.109 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:34.109 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:34.109 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:34.109 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2024-03-29T11:18:34.109 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:34.110 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 964 k 2024-03-29T11:18:34.110 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 2.8 M 2024-03-29T11:18:34.110 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:34.241 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 603 kB/s | 79 kB 00:00 2024-03-29T11:18:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:34.394 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:18:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:34.394 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:34.524 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.1 MB/s | 885 kB 00:00 2024-03-29T11:18:34.525 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:34.525 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.3 MB/s | 964 kB 00:00 2024-03-29T11:18:34.525 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:34.547 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:34.547 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:34.624 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:34.624 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:34.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs2 2024-03-29T11:18:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:34.954 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:35.043 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:35.253 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:05 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:35.354 INFO:teuthology.orchestra.run.smithi172.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:35.410 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:35.411 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-29T11:18:35.411 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:35.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:35.711 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:35.711 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:36.021 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:36.021 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:36.021 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:36.021 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:36.021 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:36.021 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:36.062 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:06 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:36.214 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:36.215 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:36.215 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-03-29T11:18:36.215 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:36.215 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:18:36.215 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:36.215 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:36.216 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:18:36.216 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:36.216 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-29T11:18:36.216 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:36.216 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 31 k 2024-03-29T11:18:36.216 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 140 k 2024-03-29T11:18:36.216 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:18:36.328 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-volume 2024-03-29T11:18:36.451 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 132 kB/s | 31 kB 00:00 2024-03-29T11:18:36.452 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:36.452 INFO:teuthology.orchestra.run.smithi172.stdout:Total 132 kB/s | 31 kB 00:00 2024-03-29T11:18:36.452 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:18:36.456 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:18:36.456 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:18:36.480 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:18:36.481 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:18:36.606 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:18:36.716 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:36.914 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:06 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:37.072 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:37.073 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-29T11:18:37.074 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:37.074 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:37.074 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-29T11:18:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:37.074 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:37.075 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:37.075 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2024-03-29T11:18:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:37.075 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 332 k 2024-03-29T11:18:37.075 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 1.5 M 2024-03-29T11:18:37.075 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:37.196 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 573 kB/s | 69 kB 00:00 2024-03-29T11:18:37.357 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:37.357 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:37.357 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:18:37.357 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:37.357 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:37.357 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:37.478 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 654 kB/s | 264 kB 00:00 2024-03-29T11:18:37.479 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:37.479 INFO:teuthology.orchestra.run.smithi059.stdout:Total 819 kB/s | 332 kB 00:00 2024-03-29T11:18:37.479 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:37.489 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:37.489 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:37.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados2 2024-03-29T11:18:37.529 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:37.529 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:37.710 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:37.885 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:37.922 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:37.938 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:37.938 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:38.090 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:08 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:38.192 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:38.245 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:38.246 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-29T11:18:38.246 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:38.298 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librbd1 2024-03-29T11:18:38.675 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:38.898 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:08 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:39.000 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:39.037 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:39.037 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:39.037 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:39.037 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:39.037 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:39.037 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:39.037 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:39.054 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:39.055 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-29T11:18:39.055 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:39.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rados 2024-03-29T11:18:39.346 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel 2024-03-29T11:18:39.676 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:09 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:39.777 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:39.829 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:39.830 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-29T11:18:39.830 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:39.882 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rgw 2024-03-29T11:18:39.922 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:09 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:40.078 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:40.079 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:40.079 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-03-29T11:18:40.079 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:40.079 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:40.079 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:40.079 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:40.080 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:40.080 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:40.080 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-29T11:18:40.080 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:40.080 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 127 k 2024-03-29T11:18:40.080 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 456 k 2024-03-29T11:18:40.080 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:40.450 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:40.460 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 333 kB/s | 127 kB 00:00 2024-03-29T11:18:40.460 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:40.461 INFO:teuthology.orchestra.run.smithi059.stdout:Total 332 kB/s | 127 kB 00:00 2024-03-29T11:18:40.461 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:40.467 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:40.467 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:40.526 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:40.526 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:40.548 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:40.601 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:40.602 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-29T11:18:40.602 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:40.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-cephfs 2024-03-29T11:18:40.679 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:40.786 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:41.214 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:41.226 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:11 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:41.324 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:41.376 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:41.377 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-29T11:18:41.378 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:41.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rbd 2024-03-29T11:18:41.527 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:41.528 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:41.528 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:41.528 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:41.528 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:41.528 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:41.757 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2 2024-03-29T11:18:42.000 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:12 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:42.098 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:42.150 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:42.151 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-29T11:18:42.151 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:42.202 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-fuse 2024-03-29T11:18:42.337 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:12 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:42.439 INFO:teuthology.orchestra.run.smithi059.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:42.497 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:42.498 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-29T11:18:42.499 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:42.579 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:42.766 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:12 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:42.915 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 85 k 2024-03-29T11:18:42.916 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 231 k 2024-03-29T11:18:42.917 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:18:43.160 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:13 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:43.246 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 259 kB/s | 85 kB 00:00 2024-03-29T11:18:43.247 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:43.248 INFO:teuthology.orchestra.run.smithi172.stdout:Total 257 kB/s | 85 kB 00:00 2024-03-29T11:18:43.248 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:18:43.255 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:18:43.255 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:18:43.310 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:18:43.310 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:18:43.321 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:43.321 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:43.321 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-03-29T11:18:43.321 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:43.322 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:43.322 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:43.322 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:43.322 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:43.322 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:43.322 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-29T11:18:43.322 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:43.323 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 31 k 2024-03-29T11:18:43.323 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 140 k 2024-03-29T11:18:43.323 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:43.449 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:18:43.488 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 188 kB/s | 31 kB 00:00 2024-03-29T11:18:43.489 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:43.489 INFO:teuthology.orchestra.run.smithi059.stdout:Total 187 kB/s | 31 kB 00:00 2024-03-29T11:18:43.489 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:43.492 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:43.492 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:43.529 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:43.530 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:43.553 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:43.672 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:43.782 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:43.977 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:44.152 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:44.306 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:44.306 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:44.306 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:18:44.306 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:44.306 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:44.306 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:44.428 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:44.428 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:44.428 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:44.428 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:44.428 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:44.428 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:44.538 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-mirror 2024-03-29T11:18:44.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2 2024-03-29T11:18:45.117 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:45.165 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:45.265 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:45.273 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:45.273 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:45.273 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-03-29T11:18:45.274 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:45.274 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:18:45.274 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:45.274 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:45.274 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:18:45.274 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:45.274 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-29T11:18:45.274 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:45.275 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 3.1 M 2024-03-29T11:18:45.275 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 13 M 2024-03-29T11:18:45.275 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:18:45.320 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:45.321 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-29T11:18:45.321 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:45.371 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1 2024-03-29T11:18:45.741 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 6.6 MB/s | 3.1 MB 00:00 2024-03-29T11:18:45.741 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:45.742 INFO:teuthology.orchestra.run.smithi172.stdout:Total 6.6 MB/s | 3.1 MB 00:00 2024-03-29T11:18:45.742 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:18:45.751 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:18:45.751 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:18:45.826 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:18:45.827 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:18:45.946 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:46.051 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:46.107 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:46.108 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-29T11:18:46.108 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:46.160 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados 2024-03-29T11:18:46.246 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:18:46.300 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:46.325 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:46.325 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:46.326 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:46.326 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:46.326 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:46.326 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:46.732 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:46.834 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:46.889 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-29T11:18:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:46.946 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw 2024-03-29T11:18:47.440 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:47.440 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:47.440 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:18:47.440 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:47.440 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:47.440 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:47.526 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:47.628 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:47.683 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:47.684 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-29T11:18:47.684 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:47.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-nbd 2024-03-29T11:18:47.739 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs 2024-03-29T11:18:48.284 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:48.318 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:48.418 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:48.433 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-29T11:18:48.433 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:48.433 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:48.433 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:48.433 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-29T11:18:48.433 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:48.433 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:48.434 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-29T11:18:48.434 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-29T11:18:48.434 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-29T11:18:48.434 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:48.434 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 171 k 2024-03-29T11:18:48.434 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 494 k 2024-03-29T11:18:48.435 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-29T11:18:48.473 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:48.474 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-29T11:18:48.474 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:48.531 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd 2024-03-29T11:18:49.011 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 298 kB/s | 171 kB 00:00 2024-03-29T11:18:49.012 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:49.012 INFO:teuthology.orchestra.run.smithi172.stdout:Total 297 kB/s | 171 kB 00:00 2024-03-29T11:18:49.012 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-29T11:18:49.021 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-29T11:18:49.021 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-29T11:18:49.080 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-29T11:18:49.080 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-29T11:18:49.109 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:49.211 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-29T11:18:49.266 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:49.267 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-29T11:18:49.267 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:49.322 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse 2024-03-29T11:18:49.334 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:49.895 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:50.039 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:50.039 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:50.040 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-29T11:18:50.040 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:50.040 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:18:50.040 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-29T11:18:50.049 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:50.049 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:50.050 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:50.050 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:50.050 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:50.050 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:50.050 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:50.050 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:50.050 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:50.050 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-29T11:18:50.050 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:50.051 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 85 k 2024-03-29T11:18:50.051 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 231 k 2024-03-29T11:18:50.051 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:50.281 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:50.365 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 271 kB/s | 85 kB 00:00 2024-03-29T11:18:50.366 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:50.366 INFO:teuthology.orchestra.run.smithi059.stdout:Total 270 kB/s | 85 kB 00:00 2024-03-29T11:18:50.366 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:50.374 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:50.374 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:50.433 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:50.433 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:50.579 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:50.692 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:51.076 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:51.378 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:51.378 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:51.378 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:51.378 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:51.378 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:51.378 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:51.609 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2024-03-29T11:18:52.186 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:52.344 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:52.345 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-03-29T11:18:52.345 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:52.345 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:52.345 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:52.345 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:52.346 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-29T11:18:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:52.346 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 3.1 M 2024-03-29T11:18:52.346 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 13 M 2024-03-29T11:18:52.346 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:54.234 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.6 MB/s | 3.1 MB 00:01 2024-03-29T11:18:54.235 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:54.235 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.6 MB/s | 3.1 MB 00:01 2024-03-29T11:18:54.236 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:54.329 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:54.330 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:54.788 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:54.860 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:54.884 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:54.885 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:54.885 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:54.885 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:56.034 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:56.034 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:56.034 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:56.034 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:56.034 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:56.035 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:56.299 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd 2024-03-29T11:18:56.879 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 11:16:30 AM UTC. 2024-03-29T11:18:57.035 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-29T11:18:57.036 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:57.036 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:57.036 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:57.036 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-29T11:18:57.036 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:57.036 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:57.036 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-29T11:18:57.037 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-29T11:18:57.037 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-29T11:18:57.037 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:57.037 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 171 k 2024-03-29T11:18:57.037 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 494 k 2024-03-29T11:18:57.037 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-29T11:18:57.495 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 374 kB/s | 171 kB 00:00 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi059.stdout:Total 373 kB/s | 171 kB 00:00 2024-03-29T11:18:57.496 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-29T11:18:57.505 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-29T11:18:57.564 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-29T11:18:57.564 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-29T11:18:57.720 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-29T11:18:57.822 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:58.243 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-29T11:18:58.526 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:18:58.527 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-29T11:18:58.781 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:58.782 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:58.920 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:58.941 INFO:teuthology.orchestra.run.smithi059.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:58.941 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:58.941 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:58.943 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:59.079 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:59.104 INFO:teuthology.orchestra.run.smithi172.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:59.105 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:59.105 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:59.106 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:18:59.107 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:18:59.107 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:59.134 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:18:59.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:59.176 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:18:59.177 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:18:59.177 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:59.204 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:59.270 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:18:59.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:59.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:59.368 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:18:59.368 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:18:59.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:59.398 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:59.465 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:18:59.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:59.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:59.562 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:18:59.562 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:18:59.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:59.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:59.656 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:18:59.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:59.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:59.750 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:18:59.834 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:18:59.835 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:59.837 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:59.864 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:18:59.864 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:59.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:59.930 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:18:59.930 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:59.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:19:00.008 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:19:00.008 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:19:00.008 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622689', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi059.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpM4AhB3Xc7/K7D8W6puQ5pHgaOxyi5wEP+yhHj9ifzq6krw/XMR2L09TlXLiV16kErqxR0s9oT5hpN8e/DsHQ=', 'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLKHThZ4r0Bb1ok8Y8VwtGYVX4plSSApvkvHIR3vWr7lFP6SSIFvSZm56wznkB3V50jsyn0stxgaz0jPKH27bzk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:19:00.009 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:19:00.038 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:19:00.053 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:19:00.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:19:00.120 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:00.120 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:19:00.171 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:19:00.171 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-03-29T11:19:00.225 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:19:00.226 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:00.226 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:19:00.226 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:00.226 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:00.226 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-29 11:18:55.169043739 +0000 2024-03-29T11:19:00.226 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-29 11:15:41.336924523 +0000 2024-03-29T11:19:00.226 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-29 11:15:41.336924523 +0000 2024-03-29T11:19:00.226 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-29 11:15:41.336924523 +0000 2024-03-29T11:19:00.226 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:19:00.290 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-29T11:19:00.290 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-29T11:19:00.290 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000204139 s, 2.5 MB/s 2024-03-29T11:19:00.291 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:19:00.346 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-03-29T11:19:00.400 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:19:00.401 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:00.401 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:19:00.401 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:00.401 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:00.401 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-29 11:18:55.170043727 +0000 2024-03-29T11:19:00.401 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-29 11:15:41.674920192 +0000 2024-03-29T11:19:00.401 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-29 11:15:41.674920192 +0000 2024-03-29T11:19:00.401 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-29 11:15:41.674920192 +0000 2024-03-29T11:19:00.401 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:19:00.465 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-29T11:19:00.465 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-29T11:19:00.466 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000187214 s, 2.7 MB/s 2024-03-29T11:19:00.466 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:19:00.522 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-03-29T11:19:00.576 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:19:00.576 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:00.577 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:19:00.577 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:00.577 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:00.577 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-29 11:18:55.170043727 +0000 2024-03-29T11:19:00.577 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-29 11:15:42.022915734 +0000 2024-03-29T11:19:00.577 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-29 11:15:42.022915734 +0000 2024-03-29T11:19:00.577 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-29 11:15:42.022915734 +0000 2024-03-29T11:19:00.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:19:00.641 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-29T11:19:00.641 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-29T11:19:00.641 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000182097 s, 2.8 MB/s 2024-03-29T11:19:00.642 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:19:00.697 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-03-29T11:19:00.751 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:19:00.751 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:00.751 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:19:00.752 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:00.752 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:00.752 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-29 11:18:55.171043714 +0000 2024-03-29T11:19:00.752 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-29 11:15:42.386911070 +0000 2024-03-29T11:19:00.752 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-29 11:15:42.386911070 +0000 2024-03-29T11:19:00.752 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-29 11:15:42.386911070 +0000 2024-03-29T11:19:00.752 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:19:00.816 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-29T11:19:00.816 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-29T11:19:00.816 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000187798 s, 2.7 MB/s 2024-03-29T11:19:00.817 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:19:00.872 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:00.873 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:00.873 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:19:00.890 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:19:00.890 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-03-29T11:19:00.946 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:19:00.947 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:00.947 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:19:00.947 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:00.947 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:00.947 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-29 11:18:46.581128326 +0000 2024-03-29T11:19:00.947 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-29 11:15:41.313686273 +0000 2024-03-29T11:19:00.947 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-29 11:15:41.313686273 +0000 2024-03-29T11:19:00.947 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-29 11:15:41.313686273 +0000 2024-03-29T11:19:00.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:19:01.014 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-29T11:19:01.014 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-29T11:19:01.015 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000213106 s, 2.4 MB/s 2024-03-29T11:19:01.016 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:19:01.073 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-03-29T11:19:01.129 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:19:01.129 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:01.129 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:19:01.129 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:01.130 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:01.130 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-29 11:18:46.581128326 +0000 2024-03-29T11:19:01.130 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-29 11:15:41.657678037 +0000 2024-03-29T11:19:01.130 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-29 11:15:41.657678037 +0000 2024-03-29T11:19:01.130 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-29 11:15:41.657678037 +0000 2024-03-29T11:19:01.130 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:19:01.196 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-29T11:19:01.197 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-29T11:19:01.197 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000194527 s, 2.6 MB/s 2024-03-29T11:19:01.198 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:19:01.255 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-03-29T11:19:01.312 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:19:01.312 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:01.312 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:19:01.312 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:01.312 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:01.312 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-29 11:18:46.582128302 +0000 2024-03-29T11:19:01.312 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-29 11:15:41.994669968 +0000 2024-03-29T11:19:01.312 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-29 11:15:41.994669968 +0000 2024-03-29T11:19:01.312 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-29 11:15:41.994669968 +0000 2024-03-29T11:19:01.313 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:19:01.379 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-29T11:19:01.379 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-29T11:19:01.379 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000207381 s, 2.5 MB/s 2024-03-29T11:19:01.381 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:19:01.437 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-03-29T11:19:01.493 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:19:01.493 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:01.493 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:19:01.493 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:01.494 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:01.494 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-29 11:18:46.582128302 +0000 2024-03-29T11:19:01.494 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-29 11:15:42.334661826 +0000 2024-03-29T11:19:01.494 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-29 11:15:42.334661826 +0000 2024-03-29T11:19:01.494 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-29 11:15:42.334661826 +0000 2024-03-29T11:19:01.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:19:01.559 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-29T11:19:01.559 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-29T11:19:01.560 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000163277 s, 3.1 MB/s 2024-03-29T11:19:01.560 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:19:01.617 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:01.617 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:19:01.617 INFO:tasks.ceph:Generating config... 2024-03-29T11:19:01.618 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:19:01.618 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:19:01.618 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:19:01.618 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:19:01.618 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:19:01.618 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:19:01.619 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:19:01.619 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:19:01.619 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:19:01.619 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:19:01.619 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:19:01.619 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:19:01.619 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:19:01.619 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:19:01.619 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:19:01.619 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:19:01.619 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:19:01.619 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:19:01.619 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:19:01.619 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:19:01.620 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:19:01.620 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:19:01.620 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:19:01.621 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:19:01.621 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:19:01.622 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:01.673 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:19:01.675 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:19:01.722 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:01.748 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.59:6789]'), ('mon.c', '[v1:172.21.15.59:6790]'), ('mon.b', '[v1:172.21.15.172:6789]')] 2024-03-29T11:19:01.749 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.59:6789],[v1:172.21.15.59:6790],[v1:172.21.15.172:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:19:01.750 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:01.750 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:19:01.803 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.59:6789]' --addv c '[v1:172.21.15.59:6790]' --addv b '[v1:172.21.15.172:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:01.888 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: generated fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = quincy 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 0 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:min_mon_release 17 (quincy) 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:election_strategy: 1 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:01.889 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:19:01.890 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:19:01.904 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6661e61e-58ea-43e3-8b64-266f0a1dca60... 2024-03-29T11:19:01.905 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:19:01.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:19:01.994 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-03-29T11:19:01.994 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-03-29T11:19:01.994 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:19:01.994 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-03-29T11:19:01.995 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-03-29T11:19:01.996 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "[v1:172.21.15.59:6789],[v1:172.21.15.59:6790],[v1:172.21.15.172:6789]" 2024-03-29T11:19:01.997 INFO:teuthology.orchestra.run.smithi059.stdout: lockdep = True 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: mon client directed command retry = 5 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: mon election default strategy = 1 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: ms bind msgr2 = False 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: ms inject socket failures = 5000 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: ms type = async 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-03-29T11:19:01.998 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-03-29T11:19:01.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-03-29T11:19:02.000 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore compression algorithm = snappy 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore compression mode = aggressive 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore zero block detection = True 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 20 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 20 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: debug monc = 20 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 10 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: osd class default list = * 2024-03-29T11:19:02.001 INFO:teuthology.orchestra.run.smithi059.stdout: osd class load list = * 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd client watch timeout = 120 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify cached snaps = True 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify missing on start = True 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock override recovery settings = True 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd op queue = debug_random 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd op queue cut off = debug_random 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd shutdown pgref assert = True 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-29T11:19:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-03-29T11:19:02.003 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:19:02.004 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: mon min osdmap epochs = 50 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune interval = 2 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune min = 15 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: mon scrub interval = 300 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on pool no app = False 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: paxos service trim min = 10 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:19:02.005 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:19:02.006 INFO:teuthology.orchestra.run.smithi059.stdout: debug monc = 20 2024-03-29T11:19:02.006 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-29T11:19:02.006 INFO:teuthology.orchestra.run.smithi059.stdout: debug objecter = 20 2024-03-29T11:19:02.006 INFO:teuthology.orchestra.run.smithi059.stdout: debug rados = 20 2024-03-29T11:19:02.006 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-03-29T11:19:02.006 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-03-29T11:19:02.006 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-03-29T11:19:02.006 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2024-03-29T11:19:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:19:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2024-03-29T11:19:02.009 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = "[v1:172.21.15.59:6789],[v1:172.21.15.59:6790],[v1:172.21.15.172:6789]" 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: lockdep = True 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: mon client directed command retry = 5 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: mon election default strategy = 1 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: ms bind msgr2 = False 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject socket failures = 5000 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: ms type = async 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2024-03-29T11:19:02.010 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug trim objects = true 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.011 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression algorithm = snappy 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression mode = aggressive 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2024-03-29T11:19:02.012 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore zero block detection = True 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 20 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 20 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: debug monc = 20 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 10 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: osd class default list = * 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: osd class load list = * 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: osd client watch timeout = 120 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify cached snaps = True 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify missing on start = True 2024-03-29T11:19:02.013 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock override recovery settings = True 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: osd op queue = debug_random 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: osd op queue cut off = debug_random 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: osd shutdown pgref assert = True 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:02.014 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.015 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: mon min osdmap epochs = 50 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune interval = 2 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune min = 15 2024-03-29T11:19:02.016 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: mon scrub interval = 300 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on pool no app = False 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: paxos service trim min = 10 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: debug monc = 20 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: debug objecter = 20 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout: debug rados = 20 2024-03-29T11:19:02.017 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2024-03-29T11:19:02.018 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.c] 2024-03-29T11:19:02.018 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.b] 2024-03-29T11:19:02.019 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:19:02.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:19:02.074 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:19:02.074 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:02.074 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:19:02.088 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:02.088 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:19:02.142 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi059.front.sepia.ceph.com 2024-03-29T11:19:02.143 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:02.143 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:19:02.143 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:02.221 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:02.221 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:02.277 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2024-03-29T11:19:02.277 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:02.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:19:02.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:02.316 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:02.317 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:02.372 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:19:02.372 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:19:02.372 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:19:02.428 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:19:02.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:19:02.495 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:19:02.498 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:19:02.498 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:19:02.498 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:19:02.540 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:19:02.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:19:02.598 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:19:02.613 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:19:02.614 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:19:02.614 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:02.640 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:02.640 INFO:tasks.ceph:role: osd.0 2024-03-29T11:19:02.640 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-29T11:19:02.641 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:19:02.714 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:02.714 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:02.714 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:02.714 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:02.714 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:02.714 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:02.715 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:02.715 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:02.715 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:02.715 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:03.208 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-29T11:19:03.210 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-29T11:19:03.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:03.329 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:03.360 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:03.426 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:03.427 INFO:tasks.ceph:role: osd.1 2024-03-29T11:19:03.427 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-29T11:19:03.427 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:19:03.500 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:03.500 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:03.500 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:03.500 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:03.500 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:03.501 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:03.501 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:03.501 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:03.501 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:03.501 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:03.961 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-29T11:19:03.963 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-29T11:19:03.963 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:04.064 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:04.095 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:04.163 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:04.163 INFO:tasks.ceph:role: osd.2 2024-03-29T11:19:04.163 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-29T11:19:04.163 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:19:04.236 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:04.236 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:04.237 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:04.237 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:04.237 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:04.237 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:04.237 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:04.237 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:04.237 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:04.237 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:04.719 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-29T11:19:04.721 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-29T11:19:04.722 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:04.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:04.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:04.939 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:04.939 INFO:tasks.ceph:role: osd.3 2024-03-29T11:19:04.939 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-29T11:19:04.940 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:19:05.014 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:05.014 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:05.015 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:05.015 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:05.015 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:05.015 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:05.015 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:05.015 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:05.015 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:05.015 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:05.537 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-29T11:19:05.539 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-29T11:19:05.539 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:05.670 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:05.704 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:05.823+0000 7fbd35b49640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:19:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:05.824+0000 7fbd35b49640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:19:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:05.824+0000 7fbd35b49640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:05.824+0000 7fbd35b49640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:05.824+0000 7fbd35b49640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:05.824+0000 7fbd35b49640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:19:08.648 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:08.678 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:08.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:08.792+0000 7fc0acc8b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:19:08.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:08.792+0000 7fc0acc8b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:19:08.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:08.792+0000 7fc0acc8b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:08.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:08.792+0000 7fc0acc8b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:08.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:08.792+0000 7fc0acc8b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:08.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:08.793+0000 7fc0acc8b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:19:11.623 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:11.655 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:11.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:11.768+0000 7f484b624640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:19:11.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:11.769+0000 7f484b624640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:19:11.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:11.769+0000 7f484b624640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:11.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:11.769+0000 7f484b624640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:11.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:11.769+0000 7f484b624640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:11.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:11.769+0000 7f484b624640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:19:14.596 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:14.627 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:14.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:14.740+0000 7fa380820640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:19:14.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:14.741+0000 7fa380820640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:19:14.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:14.741+0000 7fa380820640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:14.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:14.741+0000 7fa380820640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:14.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:14.741+0000 7fa380820640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:14.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:14.741+0000 7fa380820640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:19:17.568 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:17.599 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:17.631 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:17.631 INFO:tasks.ceph:role: osd.4 2024-03-29T11:19:17.631 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-29T11:19:17.631 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:19:17.708 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:17.709 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:17.709 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:17.709 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:17.709 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:17.709 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:17.709 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:17.709 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:17.709 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:17.709 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:18.228 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-29T11:19:18.231 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-29T11:19:18.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:18.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:18.418 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:18.486 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:18.486 INFO:tasks.ceph:role: osd.5 2024-03-29T11:19:18.486 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-29T11:19:18.486 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:19:18.561 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:18.562 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:18.562 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:18.562 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:18.562 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:18.562 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:18.562 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:18.562 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:18.562 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:18.562 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:19.012 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-29T11:19:19.015 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-29T11:19:19.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:19.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:19.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:19.213 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:19.213 INFO:tasks.ceph:role: osd.6 2024-03-29T11:19:19.213 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-29T11:19:19.214 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:19:19.289 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:19.289 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:19.289 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:19.289 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:19.289 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:19.785 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-29T11:19:19.787 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-29T11:19:19.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:19.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:19.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:20.045 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:20.045 INFO:tasks.ceph:role: osd.7 2024-03-29T11:19:20.045 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-29T11:19:20.045 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:19:20.119 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:20.119 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:20.119 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:20.119 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:20.119 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:20.630 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-29T11:19:20.631 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-29T11:19:20.632 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:20.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:20.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:20.927+0000 7f0aff0f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:20.927+0000 7f0aff0f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:20.927+0000 7f0aff0f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:20.927+0000 7f0aff0f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:20.927+0000 7f0aff0f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:20.927+0000 7f0aff0f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:19:23.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:23.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:23.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:23.902+0000 7fc0b020f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:19:23.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:23.902+0000 7fc0b020f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:19:23.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:23.902+0000 7fc0b020f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:23.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:23.902+0000 7fc0b020f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:23.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:23.902+0000 7fc0b020f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:23.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:23.903+0000 7fc0b020f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:19:26.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:26.762 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:26.874+0000 7f2f723ac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:26.874+0000 7f2f723ac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:26.874+0000 7f2f723ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:26.874+0000 7f2f723ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:26.874+0000 7f2f723ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:26.875+0000 7f2f723ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:19:29.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:29.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:29.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:29.851+0000 7fe8609df640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:19:29.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:29.851+0000 7fe8609df640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:19:29.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:29.851+0000 7fe8609df640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:29.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:29.851+0000 7fe8609df640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:29.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:29.851+0000 7fe8609df640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:29.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:29.852+0000 7fe8609df640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:19:32.673 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:32.704 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:19:32.705 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:32.705 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:19:32.736 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:32.736 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:19:32.807 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:32.807 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:19:32.876 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:32.876 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:19:32.944 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:32.944 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:19:33.013 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:33.013 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:19:33.044 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:33.044 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:19:33.114 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:33.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:19:33.185 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:33.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:19:33.254 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:33.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:19:33.323 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:33.323 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:19:33.340 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:33.340 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:19:33.379 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:19:33.379 DEBUG:teuthology.orchestra.run.smithi059:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:33.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:33.422 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.y] 2024-03-29T11:19:33.422 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCmowZmUBX9GhAAzCiWVc2eb96Zzpd0N0G9iw== 2024-03-29T11:19:33.422 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2024-03-29T11:19:33.422 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCpowZmeyc2MRAA0U0A6/kKTfgoaF5QRYcdPA== 2024-03-29T11:19:33.422 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2024-03-29T11:19:33.422 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCsowZmpYVWLxAA85Od6AmsosllNfnLjroF7A== 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCvowZmRDjsLRAA3exSaKwRwhb2veRKeyNy1g== 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.3] 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCyowZm30ZFLBAAhJbdcjPsHbaD1ZkBXKNG8A== 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.x] 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCmowZmtGT9HhAAZ3mgMj9znG4X2OHH6c8lgw== 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.4] 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQC4owZmSHtINxAAbn+vhuzjQH5PG+aKOzEmNg== 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.5] 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQC7owZm61vNNRAA6L82Lif0IR/Nm0s6sBqiKA== 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.6] 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQC+owZmviolNBAAPuPUbpVfN+K0JmfgGkmNug== 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.7] 2024-03-29T11:19:33.423 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDBowZmSyLFMhAAYF3Dv7wu0Ekn5e/l6YmQuA== 2024-03-29T11:19:33.424 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-29T11:19:33.424 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCmowZmYI2nIRAAUTNeWFq1xlzsvMTgJaI07Q== 2024-03-29T11:19:33.424 INFO:teuthology.orchestra.run.smithi059.stdout:[client.1] 2024-03-29T11:19:33.424 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCmowZmjBQfJRAAzUdt3VqMIATarRmIZtaGvQ== 2024-03-29T11:19:33.446 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.y] 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCmowZmUBX9GhAAzCiWVc2eb96Zzpd0N0G9iw== 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCpowZmeyc2MRAA0U0A6/kKTfgoaF5QRYcdPA== 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCsowZmpYVWLxAA85Od6AmsosllNfnLjroF7A== 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCvowZmRDjsLRAA3exSaKwRwhb2veRKeyNy1g== 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.3] 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCyowZm30ZFLBAAhJbdcjPsHbaD1ZkBXKNG8A== 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCmowZmtGT9HhAAZ3mgMj9znG4X2OHH6c8lgw== 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.4] 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQC4owZmSHtINxAAbn+vhuzjQH5PG+aKOzEmNg== 2024-03-29T11:19:33.447 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.5] 2024-03-29T11:19:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQC7owZm61vNNRAA6L82Lif0IR/Nm0s6sBqiKA== 2024-03-29T11:19:33.448 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.6] 2024-03-29T11:19:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQC+owZmviolNBAAPuPUbpVfN+K0JmfgGkmNug== 2024-03-29T11:19:33.448 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.7] 2024-03-29T11:19:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDBowZmSyLFMhAAYF3Dv7wu0Ekn5e/l6YmQuA== 2024-03-29T11:19:33.448 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2024-03-29T11:19:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCmowZmYI2nIRAAUTNeWFq1xlzsvMTgJaI07Q== 2024-03-29T11:19:33.448 INFO:teuthology.orchestra.run.smithi172.stdout:[client.1] 2024-03-29T11:19:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCmowZmjBQfJRAAzUdt3VqMIATarRmIZtaGvQ== 2024-03-29T11:19:33.449 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:33.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:33.543 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.598 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.653 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.708 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.763 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:33.765 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:33.817 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.925 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.927 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.979 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:33.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:34.034 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:34.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:34.091 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:34.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:34.148 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:19:34.148 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:34.174 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:34.270 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:34.300 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:34.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:34.462 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:34.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:34.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:34.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:34.640 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:34.642 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:34.694 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:19:34.695 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:19:34.695 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:19:34.697 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:19:34.697 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:19:34.697 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:19:34.699 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:19:34.699 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:19:34.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:19:34.737 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:19:34.738 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:19:34.738 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:19:34.738 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:19:34.740 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:19:34.740 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:19:34.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:19:34.743 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:19:34.743 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:19:34.743 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.868+0000 7f7b4bc25640 10 monclient: get_monmap_and_config 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.868+0000 7f7b4bc25640 10 monclient: build_initial_monmap 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.868+0000 7f7b4bc25640 10 monclient: monmap: 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:34.869277+0000 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:34.869277+0000 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:34.845 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 1 Processor -- start 2024-03-29T11:19:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 1 -- start start 2024-03-29T11:19:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 10 monclient: init 2024-03-29T11:19:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 10 monclient(hunting): picked mon.noname-a con 0x7f7b4409ac40 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 10 monclient(hunting): picked mon.noname-c con 0x7f7b4409c870 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 10 monclient(hunting): picked mon.noname-b con 0x7f7b440b4360 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4409dec0 con 0x7f7b4409ac40 2024-03-29T11:19:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4409da30 con 0x7f7b4409c870 2024-03-29T11:19:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b440b39c0 con 0x7f7b440b4360 2024-03-29T11:19:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4bc25640 10 monclient(hunting): authenticate will time out at 824.983551s 2024-03-29T11:19:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b499c1640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7b4409ac40 0x7f7b440b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50740/0 (socket says 172.21.15.59:50740) 2024-03-29T11:19:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b4a1c2640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f7b440b4360 0x7f7b4414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:47568/0 (socket says 172.21.15.59:47568) 2024-03-29T11:19:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:34.869+0000 7f7b499c1640 1 -- 172.21.15.59:0/689538244 learned_addr learned my addr 172.21.15.59:0/689538244 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:19:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 10 monclient(hunting): tick 2024-03-29T11:19:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:19:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 1 monclient(hunting): continuing hunt 2024-03-29T11:19:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:19:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 1 -- 172.21.15.59:0/689538244 >> v1:172.21.15.59:6790/0 conn(0x7f7b440b4360 legacy=0x7f7b4414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 1 -- 172.21.15.59:0/689538244 >> v1:172.21.15.172:6789/0 conn(0x7f7b4409c870 legacy=0x7f7b4413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 1 -- 172.21.15.59:0/689538244 >> v1:172.21.15.59:6789/0 conn(0x7f7b4409ac40 legacy=0x7f7b440b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f7b280036e0 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f7b280072d0 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f7b2800aff0 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:37.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 1 -- 172.21.15.59:0/689538244 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b440b5160 con 0x7f7b280072d0 2024-03-29T11:19:37.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 1 -- 172.21.15.59:0/689538244 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2800fc80 con 0x7f7b280036e0 2024-03-29T11:19:37.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 1 -- 172.21.15.59:0/689538244 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b28010ee0 con 0x7f7b2800aff0 2024-03-29T11:19:37.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:37.870+0000 7f7b3b7fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:40.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.037+0000 7f7b489bf640 1 -- 172.21.15.59:0/689538244 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139719402 0 0) 0x7f7b440b5160 con 0x7f7b280072d0 2024-03-29T11:19:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.037+0000 7f7b489bf640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.037+0000 7f7b489bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.037+0000 7f7b489bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.037+0000 7f7b489bf640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:19:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.037+0000 7f7b489bf640 1 -- 172.21.15.59:0/689538244 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b30003520 con 0x7f7b280072d0 2024-03-29T11:19:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 1 -- 172.21.15.59:0/689538244 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465971468 0 0) 0x7f7b2800fc80 con 0x7f7b280036e0 2024-03-29T11:19:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:19:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 1 -- 172.21.15.59:0/689538244 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b440b5160 con 0x7f7b280036e0 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 1 -- 172.21.15.59:0/689538244 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1940303172 0 0) 0x7f7b440b5160 con 0x7f7b280036e0 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 1 -- 172.21.15.59:0/689538244 >> v1:172.21.15.59:6790/0 conn(0x7f7b2800aff0 legacy=0x7f7b2800d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 1 -- 172.21.15.59:0/689538244 >> v1:172.21.15.59:6789/0 conn(0x7f7b280072d0 legacy=0x7f7b28009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 10 monclient: _finish_hunting 0 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 1 monclient: found mon.noname-c 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 1 -- 172.21.15.59:0/689538244 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b28012140 con 0x7f7b280036e0 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 10 monclient: _finish_auth 0 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 10 monclient: _check_auth_tickets 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 1 -- 172.21.15.59:0/689538244 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7b400014f0 con 0x7f7b280036e0 2024-03-29T11:19:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b489bf640 10 monclient: dump: 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b4bc25640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.038+0000 7f7b4bc25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.039+0000 7f7b489bf640 1 -- 172.21.15.59:0/689538244 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b400016c0 con 0x7f7b280036e0 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.039+0000 7f7b489bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.039+0000 7f7b489bf640 1 -- 172.21.15.59:0/689538244 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7b40001a50 con 0x7f7b280036e0 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.039+0000 7f7b4bc25640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.039+0000 7f7b4bc25640 4 set_mon_vals no callback set 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.039+0000 7f7b489bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.039+0000 7f7b489bf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.039+0000 7f7b489bf640 10 monclient: dump: 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.020 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.040+0000 7f7b4bc25640 10 monclient: shutdown 2024-03-29T11:19:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.040+0000 7f7b4bc25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.040+0000 7f7b4bc25640 1 -- 172.21.15.59:0/689538244 >> v1:172.21.15.172:6789/0 conn(0x7f7b280036e0 legacy=0x7f7b28005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.040+0000 7f7b4bc25640 1 -- 172.21.15.59:0/689538244 shutdown_connections 2024-03-29T11:19:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.040+0000 7f7b4bc25640 1 -- 172.21.15.59:0/689538244 >> 172.21.15.59:0/689538244 conn(0x7f7b44066b00 msgr2=0x7f7b44066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.040+0000 7f7b4bc25640 1 -- 172.21.15.59:0/689538244 shutdown_connections 2024-03-29T11:19:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.040+0000 7f7b4bc25640 1 -- 172.21.15.59:0/689538244 wait complete. 2024-03-29T11:19:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.041+0000 7f7b4bc25640 10 monclient: build_initial_monmap 2024-03-29T11:19:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.041+0000 7f7b4bc25640 10 monclient: monmap: 2024-03-29T11:19:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.041+0000 7f7b4bc25640 1 librados: starting msgr at 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.041+0000 7f7b4bc25640 1 librados: starting objecter 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.041+0000 7f7b4bc25640 1 Processor -- start 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.041+0000 7f7b4bc25640 1 -- start start 2024-03-29T11:19:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.042+0000 7f7b4bc25640 1 librados: setting wanted keys 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.042+0000 7f7b4bc25640 1 librados: calling monclient init 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.042+0000 7f7b4bc25640 10 monclient: init 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b4bc25640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b4bc25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b4bc25640 10 monclient(hunting): picked mon.noname-a con 0x7f7b280036e0 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b4bc25640 10 monclient(hunting): picked mon.noname-b con 0x7f7b280072d0 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b4bc25640 10 monclient(hunting): picked mon.noname-c con 0x7f7b2800aff0 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b4bc25640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b441397e0 con 0x7f7b280036e0 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b4bc25640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b44145a40 con 0x7f7b280072d0 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b4bc25640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b441d72d0 con 0x7f7b2800aff0 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b4bc25640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b4bc25640 10 monclient(hunting): authenticate will time out at 830.156974s 2024-03-29T11:19:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b491c0640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7b280072d0 0x7f7b441389c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:38820/0 (socket says 172.21.15.59:38820) 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b491c0640 1 -- 172.21.15.59:0/955507668 learned_addr learned my addr 172.21.15.59:0/955507668 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 10 client.?.objecter ms_handle_connect 0x7f7b280036e0 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 10 client.?.objecter ms_handle_connect 0x7f7b280072d0 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 10 client.?.objecter ms_handle_connect 0x7f7b2800aff0 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587131511 0 0) 0x7f7b441397e0 con 0x7f7b280036e0 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.043+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b28011830 con 0x7f7b280036e0 2024-03-29T11:19:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 421248415 0 0) 0x7f7b28011830 con 0x7f7b280036e0 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b441397e0 con 0x7f7b280036e0 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7b400013f0 con 0x7f7b280036e0 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2141400252 0 0) 0x7f7b441397e0 con 0x7f7b280036e0 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 >> v1:172.21.15.59:6790/0 conn(0x7f7b2800aff0 legacy=0x7f7b441390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 >> v1:172.21.15.172:6789/0 conn(0x7f7b280072d0 legacy=0x7f7b441389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 1 monclient: found mon.a 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44146290 con 0x7f7b280036e0 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b4bc25640 5 monclient: authenticate success, global_id 4110 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b4bc25640 10 monclient: _renew_subs 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b4bc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b4bc25640 1 -- 172.21.15.59:0/955507668 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b441464e0 con 0x7f7b280036e0 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b4bc25640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b4bc25640 10 monclient: _renew_subs 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b4bc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b4bc25640 1 -- 172.21.15.59:0/955507668 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b440b1450 con 0x7f7b280036e0 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b4bc25640 1 librados: init done 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b40004a90 con 0x7f7b280036e0 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7b40004da0 con 0x7f7b280036e0 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 10 monclient: dump: 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3bfff640 4 set_mon_vals no callback set 2024-03-29T11:19:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.044+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7b40005290 con 0x7f7b280036e0 2024-03-29T11:19:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.045+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1852415274 0 0) 0x7f7b40006720 con 0x7f7b280036e0 2024-03-29T11:19:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.045+0000 7f7b3a7fc640 10 client.4110.objecter ms_dispatch 0x7f7b44000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:19:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.045+0000 7f7b3a7fc640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:19:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.045+0000 7f7b3a7fc640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:19:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.045+0000 7f7b3a7fc640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T11:19:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.046+0000 7f7b4bc25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.046+0000 7f7b4bc25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.046+0000 7f7b4bc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.046+0000 7f7b4bc25640 1 -- 172.21.15.59:0/955507668 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2c005080 con 0x7f7b280036e0 2024-03-29T11:19:40.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.050+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7b40006930 con 0x7f7b280036e0 2024-03-29T11:19:40.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.050+0000 7f7b3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.050+0000 7f7b3a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:40.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.127+0000 7f7b4bc25640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:40.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.127+0000 7f7b4bc25640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:40.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.127+0000 7f7b4bc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.127+0000 7f7b4bc25640 1 -- 172.21.15.59:0/955507668 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7b2c005330 con 0x7f7b280036e0 2024-03-29T11:19:40.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.131+0000 7f7b3a7fc640 1 -- 172.21.15.59:0/955507668 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f7b400057b0 con 0x7f7b280036e0 2024-03-29T11:19:40.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.131+0000 7f7b3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:40.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.131+0000 7f7b3a7fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:19:40.108 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2024-03-29T11:19:40.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 10 librados: watch_flush enter 2024-03-29T11:19:40.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 10 librados: watch_flush exit 2024-03-29T11:19:40.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T11:19:40.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 10 client.4110.objecter successfully canceled tick 2024-03-29T11:19:40.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 10 monclient: shutdown 2024-03-29T11:19:40.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:40.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 1 -- 172.21.15.59:0/955507668 >> v1:172.21.15.59:6789/0 conn(0x7f7b280036e0 legacy=0x7f7b441d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 1 -- 172.21.15.59:0/955507668 shutdown_connections 2024-03-29T11:19:40.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 1 -- 172.21.15.59:0/955507668 >> 172.21.15.59:0/955507668 conn(0x7f7b44066b00 msgr2=0x7f7b44069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:40.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 1 -- 172.21.15.59:0/955507668 shutdown_connections 2024-03-29T11:19:40.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 1 -- 172.21.15.59:0/955507668 wait complete. 2024-03-29T11:19:40.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:40.132+0000 7f7b4bc25640 1 librados: shutdown 2024-03-29T11:19:40.121 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:19:40.121 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:19:40.121 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:40.121 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:19:40.148 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:40.148 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:19:40.218 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:40.219 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:19:40.285 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-29T11:19:40.285 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:19:40.352 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:40.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:19:40.379 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:40.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:19:40.448 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:40.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:19:40.503 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:40.526+0000 7f1c8c80f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:40.515 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-29T11:19:40.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:19:40.518 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:40.542+0000 7fa3c7432040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:40.560 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:40.583+0000 7f1c8c80f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:40.574 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:40.598+0000 7fa3c7432040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:40.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 521f85c6-3db6-4c87-b0e3-e566a02771c1 0 2024-03-29T11:19:40.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.745+0000 7fd9a893b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.745+0000 7fd9a893b640 10 monclient: build_initial_monmap 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.745+0000 7fd9a893b640 10 monclient: monmap: 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:40.746809+0000 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:40.746809+0000 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.745+0000 7fd9a893b640 1 Processor -- start 2024-03-29T11:19:40.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.745+0000 7fd9a893b640 1 -- start start 2024-03-29T11:19:40.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.745+0000 7fd9a893b640 10 monclient: init 2024-03-29T11:19:40.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a893b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a893b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a893b640 10 monclient(hunting): picked mon.noname-c con 0x7fd9a409ac40 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a893b640 10 monclient(hunting): picked mon.noname-a con 0x7fd9a409c870 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a893b640 10 monclient(hunting): picked mon.noname-b con 0x7fd9a40b4360 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a893b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a409dec0 con 0x7fd9a409c870 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a893b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a409da30 con 0x7fd9a409ac40 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a893b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a40b39c0 con 0x7fd9a40b4360 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a893b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a893b640 10 monclient(hunting): authenticate will time out at 843.005885s 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a259c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd9a409ac40 0x7fd9a40b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44638/0 (socket says 172.21.15.172:44638) 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.746+0000 7fd9a259c640 1 -- 172.21.15.172:0/205095004 learned_addr learned my addr 172.21.15.172:0/205095004 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 -- 172.21.15.172:0/205095004 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546244945 0 0) 0x7fd9a409da30 con 0x7fd9a409ac40 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 -- 172.21.15.172:0/205095004 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd98c003520 con 0x7fd9a409ac40 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 -- 172.21.15.172:0/205095004 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464055966 0 0) 0x7fd994001110 con 0x7fd9a409c870 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 -- 172.21.15.172:0/205095004 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a409da30 con 0x7fd9a409c870 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 -- 172.21.15.172:0/205095004 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2214370549 0 0) 0x7fd98c003520 con 0x7fd9a409ac40 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 -- 172.21.15.172:0/205095004 >> v1:172.21.15.59:6790/0 conn(0x7fd9a40b4360 legacy=0x7fd9a414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 -- 172.21.15.172:0/205095004 >> v1:172.21.15.59:6789/0 conn(0x7fd9a409c870 legacy=0x7fd9a413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 monclient: found mon.noname-c 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 -- 172.21.15.172:0/205095004 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a40b5160 con 0x7fd9a409ac40 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient: _finish_auth 0 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 -- 172.21.15.172:0/205095004 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd998001690 con 0x7fd9a409ac40 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient: dump: 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a893b640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a893b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 -- 172.21.15.172:0/205095004 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd998001870 con 0x7fd9a409ac40 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 1 -- 172.21.15.172:0/205095004 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd998001cc0 con 0x7fd9a409ac40 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a893b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a893b640 4 set_mon_vals no callback set 2024-03-29T11:19:40.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a159a640 10 monclient: dump: 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a893b640 10 monclient: shutdown 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a893b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a893b640 1 -- 172.21.15.172:0/205095004 >> v1:172.21.15.172:6789/0 conn(0x7fd9a409ac40 legacy=0x7fd9a40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a893b640 1 -- 172.21.15.172:0/205095004 shutdown_connections 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.747+0000 7fd9a893b640 1 -- 172.21.15.172:0/205095004 >> 172.21.15.172:0/205095004 conn(0x7fd9a4066b00 msgr2=0x7fd9a4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.748+0000 7fd9a893b640 1 -- 172.21.15.172:0/205095004 shutdown_connections 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.748+0000 7fd9a893b640 1 -- 172.21.15.172:0/205095004 wait complete. 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.748+0000 7fd9a893b640 10 monclient: build_initial_monmap 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.748+0000 7fd9a893b640 10 monclient: monmap: 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 0.000000 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:created 0.000000 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 1 librados: starting msgr at 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 1 librados: starting objecter 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 1 Processor -- start 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 1 -- start start 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 1 librados: setting wanted keys 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 1 librados: calling monclient init 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 10 monclient: init 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 10 monclient(hunting): picked mon.noname-a con 0x7fd9a409ac40 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 10 monclient(hunting): picked mon.noname-b con 0x7fd9a4145e50 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 10 monclient(hunting): picked mon.noname-c con 0x7fd9a41dad00 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a4139b60 con 0x7fd9a409ac40 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a40ae600 con 0x7fd9a4145e50 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a40ae870 con 0x7fd9a41dad00 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:40.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a893b640 10 monclient(hunting): authenticate will time out at 843.008898s 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a1d9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd9a4145e50 0x7fd9a40b12a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44642/0 (socket says 172.21.15.172:44642) 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd9a1d9b640 1 -- 172.21.15.172:0/143454022 learned_addr learned my addr 172.21.15.172:0/143454022 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd992ffd640 10 client.?.objecter ms_handle_connect 0x7fd9a4145e50 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd992ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd992ffd640 10 client.?.objecter ms_handle_connect 0x7fd9a409ac40 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd992ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd992ffd640 10 client.?.objecter ms_handle_connect 0x7fd9a41dad00 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.749+0000 7fd992ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458882563 0 0) 0x7fd9a40ae600 con 0x7fd9a4145e50 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd984004eb0 con 0x7fd9a4145e50 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834105435 0 0) 0x7fd9a40b5160 con 0x7fd9a409ac40 2024-03-29T11:19:40.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a40ae600 con 0x7fd9a409ac40 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466395838 0 0) 0x7fd9a40ae870 con 0x7fd9a41dad00 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a40b5160 con 0x7fd9a41dad00 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4101903360 0 0) 0x7fd9a40ae600 con 0x7fd9a409ac40 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9a40ae870 con 0x7fd9a409ac40 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 277481952 0 0) 0x7fd984004eb0 con 0x7fd9a4145e50 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9a40ae600 con 0x7fd9a4145e50 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd9980019e0 con 0x7fd9a409ac40 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd9940018a0 con 0x7fd9a4145e50 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:40.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2154191841 0 0) 0x7fd9a40b5160 con 0x7fd9a41dad00 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd984004eb0 con 0x7fd9a41dad00 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd99c001a20 con 0x7fd9a41dad00 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:40.743 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1506652028 0 0) 0x7fd9a40ae870 con 0x7fd9a409ac40 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 >> v1:172.21.15.59:6790/0 conn(0x7fd9a41dad00 legacy=0x7fd9a41e2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 >> v1:172.21.15.172:6789/0 conn(0x7fd9a4145e50 legacy=0x7fd9a40b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 monclient: found mon.a 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:40.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a41da7e0 con 0x7fd9a409ac40 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd9a893b640 5 monclient: authenticate success, global_id 4116 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd9a893b640 10 monclient: _renew_subs 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd9a893b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd9a893b640 1 -- 172.21.15.172:0/143454022 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a41434a0 con 0x7fd9a409ac40 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd9a893b640 10 client.4116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd9a893b640 10 monclient: _renew_subs 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd9a893b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd9a893b640 1 -- 172.21.15.172:0/143454022 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a40b1b90 con 0x7fd9a409ac40 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd9a893b640 1 librados: init done 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd998001c40 con 0x7fd9a409ac40 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd998004d90 con 0x7fd9a409ac40 2024-03-29T11:19:40.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 monclient: dump: 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd9a0d99640 4 set_mon_vals no callback set 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd998005280 con 0x7fd9a409ac40 2024-03-29T11:19:40.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 265149116 0 0) 0x7fd998006560 con 0x7fd9a409ac40 2024-03-29T11:19:40.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 10 client.4116.objecter ms_dispatch 0x7fd9a4000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:19:40.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 3 client.4116.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:19:40.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 3 client.4116.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:19:40.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.750+0000 7fd992ffd640 20 client.4116.objecter dump_active .. 0 homeless 2024-03-29T11:19:40.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.751+0000 7fd9a893b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.751+0000 7fd9a893b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.751+0000 7fd9a893b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.751+0000 7fd9a893b640 1 -- 172.21.15.172:0/143454022 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a409b020 con 0x7fd9a409ac40 2024-03-29T11:19:40.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.753+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd998006720 con 0x7fd9a409ac40 2024-03-29T11:19:40.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.753+0000 7fd992ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.753+0000 7fd992ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:40.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.825+0000 7fd9a893b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "521f85c6-3db6-4c87-b0e3-e566a02771c1", "id": 0}] 2024-03-29T11:19:40.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.825+0000 7fd9a893b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "521f85c6-3db6-4c87-b0e3-e566a02771c1", "id": 0}] 2024-03-29T11:19:40.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.825+0000 7fd9a893b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.825+0000 7fd9a893b640 1 -- 172.21.15.172:0/143454022 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd new", "uuid": "521f85c6-3db6-4c87-b0e3-e566a02771c1", "id": 0} v 0) v1 -- 0x7fd9a413c8e0 con 0x7fd9a409ac40 2024-03-29T11:19:40.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.827+0000 7fd992ffd640 1 -- 172.21.15.172:0/143454022 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "521f85c6-3db6-4c87-b0e3-e566a02771c1", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1353023558 0 732707515) 0x7fd998006980 con 0x7fd9a409ac40 2024-03-29T11:19:40.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.827+0000 7fd992ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "521f85c6-3db6-4c87-b0e3-e566a02771c1", "id": 0}] 2024-03-29T11:19:40.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.827+0000 7fd992ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:40.805 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.827+0000 7fd9a893b640 10 librados: watch_flush enter 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.827+0000 7fd9a893b640 10 librados: watch_flush exit 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.827+0000 7fd9a893b640 20 client.4116.objecter shutdown clearing up homeless session... 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.827+0000 7fd9a893b640 10 client.4116.objecter successfully canceled tick 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.828+0000 7fd9a893b640 10 monclient: shutdown 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.828+0000 7fd9a893b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.828+0000 7fd9a893b640 1 -- 172.21.15.172:0/143454022 >> v1:172.21.15.59:6789/0 conn(0x7fd9a409ac40 legacy=0x7fd9a40ad1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.828+0000 7fd9a893b640 1 -- 172.21.15.172:0/143454022 shutdown_connections 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.828+0000 7fd9a893b640 1 -- 172.21.15.172:0/143454022 >> 172.21.15.172:0/143454022 conn(0x7fd9a4066b00 msgr2=0x7fd9a40b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.828+0000 7fd9a893b640 1 -- 172.21.15.172:0/143454022 shutdown_connections 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.828+0000 7fd9a893b640 1 -- 172.21.15.172:0/143454022 wait complete. 2024-03-29T11:19:40.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.828+0000 7fd9a893b640 1 librados: shutdown 2024-03-29T11:19:40.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new a3fb41ae-75c4-43a8-ae52-b40aa5699776 1 2024-03-29T11:19:40.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.938+0000 7f3b86bed640 10 monclient: get_monmap_and_config 2024-03-29T11:19:40.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.938+0000 7f3b86bed640 10 monclient: build_initial_monmap 2024-03-29T11:19:40.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.938+0000 7f3b86bed640 10 monclient: monmap: 2024-03-29T11:19:40.915 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:40.915 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.915 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:40.939542+0000 2024-03-29T11:19:40.915 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:40.939542+0000 2024-03-29T11:19:40.915 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:40.916 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.916 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:40.916 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:40.916 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:40.916 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.938+0000 7f3b86bed640 1 Processor -- start 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.938+0000 7f3b86bed640 1 -- start start 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.938+0000 7f3b86bed640 10 monclient: init 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b86bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b86bed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b86bed640 10 monclient(hunting): picked mon.noname-a con 0x7f3b8009ac40 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b86bed640 10 monclient(hunting): picked mon.noname-b con 0x7f3b8009c870 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b86bed640 10 monclient(hunting): picked mon.noname-c con 0x7f3b800b4360 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b86bed640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8009dec0 con 0x7f3b8009ac40 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b86bed640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8009da30 con 0x7f3b800b4360 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b86bed640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b800b39c0 con 0x7f3b8009c870 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b86bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b86bed640 10 monclient(hunting): authenticate will time out at 843.198590s 2024-03-29T11:19:40.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b8518a640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3b800b4360 0x7f3b8014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44648/0 (socket says 172.21.15.172:44648) 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b8518a640 1 -- 172.21.15.172:0/2462674348 learned_addr learned my addr 172.21.15.172:0/2462674348 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 1 -- 172.21.15.172:0/2462674348 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261389274 0 0) 0x7f3b8009da30 con 0x7f3b800b4360 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 1 -- 172.21.15.172:0/2462674348 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b70003520 con 0x7f3b800b4360 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 1 -- 172.21.15.172:0/2462674348 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1364197171 0 0) 0x7f3b70003520 con 0x7f3b800b4360 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 1 -- 172.21.15.172:0/2462674348 >> v1:172.21.15.59:6790/0 conn(0x7f3b8009c870 legacy=0x7f3b8013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 1 -- 172.21.15.172:0/2462674348 >> v1:172.21.15.59:6789/0 conn(0x7f3b8009ac40 legacy=0x7f3b800b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:40.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 1 -- 172.21.15.172:0/2462674348 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b800b5160 con 0x7f3b800b4360 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.939+0000 7f3b7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b7f7fe640 1 -- 172.21.15.172:0/2462674348 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f3b78001a10 con 0x7f3b800b4360 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b7f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b7f7fe640 10 monclient: dump: 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.927 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b7f7fe640 1 -- 172.21.15.172:0/2462674348 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b78001bf0 con 0x7f3b800b4360 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b7f7fe640 1 -- 172.21.15.172:0/2462674348 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f3b78002040 con 0x7f3b800b4360 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 4 set_mon_vals no callback set 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b7f7fe640 10 monclient: dump: 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.928 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 10 monclient: shutdown 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 1 -- 172.21.15.172:0/2462674348 >> v1:172.21.15.172:6789/0 conn(0x7f3b800b4360 legacy=0x7f3b8014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 1 -- 172.21.15.172:0/2462674348 shutdown_connections 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 1 -- 172.21.15.172:0/2462674348 >> 172.21.15.172:0/2462674348 conn(0x7f3b80066b00 msgr2=0x7f3b80066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 1 -- 172.21.15.172:0/2462674348 shutdown_connections 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.940+0000 7f3b86bed640 1 -- 172.21.15.172:0/2462674348 wait complete. 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.941+0000 7f3b86bed640 10 monclient: build_initial_monmap 2024-03-29T11:19:40.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.941+0000 7f3b86bed640 10 monclient: monmap: 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 0.000000 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:created 0.000000 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.941+0000 7f3b86bed640 1 librados: starting msgr at 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.941+0000 7f3b86bed640 1 librados: starting objecter 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.941+0000 7f3b86bed640 1 Processor -- start 2024-03-29T11:19:40.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.941+0000 7f3b86bed640 1 -- start start 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.941+0000 7f3b86bed640 1 librados: setting wanted keys 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.941+0000 7f3b86bed640 1 librados: calling monclient init 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.941+0000 7f3b86bed640 10 monclient: init 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b86bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b86bed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b86bed640 10 monclient(hunting): picked mon.noname-b con 0x7f3b8009ac40 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b86bed640 10 monclient(hunting): picked mon.noname-c con 0x7f3b800b00a0 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b86bed640 10 monclient(hunting): picked mon.noname-a con 0x7f3b800ad6f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b86bed640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b801407e0 con 0x7f3b800ad6f0 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b86bed640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b800b1430 con 0x7f3b8009ac40 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b86bed640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b801d6f40 con 0x7f3b800b00a0 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b86bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b86bed640 10 monclient(hunting): authenticate will time out at 843.201512s 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b84989640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3b8009ac40 0x7f3b800b0d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44664/0 (socket says 172.21.15.172:44664) 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b84989640 1 -- 172.21.15.172:0/4016492014 learned_addr learned my addr 172.21.15.172:0/4016492014 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:40.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 client.?.objecter ms_handle_connect 0x7f3b8009ac40 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 client.?.objecter ms_handle_connect 0x7f3b800b00a0 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 client.?.objecter ms_handle_connect 0x7f3b800ad6f0 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626816933 0 0) 0x7f3b800b1430 con 0x7f3b8009ac40 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b64004eb0 con 0x7f3b8009ac40 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126673823 0 0) 0x7f3b801407e0 con 0x7f3b800ad6f0 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b800b1430 con 0x7f3b800ad6f0 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983779791 0 0) 0x7f3b801d6f40 con 0x7f3b800b00a0 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b801407e0 con 0x7f3b800b00a0 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3082568431 0 0) 0x7f3b64004eb0 con 0x7f3b8009ac40 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b801d6f40 con 0x7f3b8009ac40 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f3b740018a0 con 0x7f3b8009ac40 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:40.933 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2422399734 0 0) 0x7f3b800b1430 con 0x7f3b800ad6f0 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b64004eb0 con 0x7f3b800ad6f0 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f3b78001f00 con 0x7f3b800ad6f0 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.942+0000 7f3b7d7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2604220701 0 0) 0x7f3b801407e0 con 0x7f3b800b00a0 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b800b1430 con 0x7f3b800b00a0 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f3b6c0018b0 con 0x7f3b800b00a0 2024-03-29T11:19:40.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3254395814 0 0) 0x7f3b64004eb0 con 0x7f3b800ad6f0 2024-03-29T11:19:40.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 >> v1:172.21.15.59:6790/0 conn(0x7f3b800b00a0 legacy=0x7f3b801d7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 >> v1:172.21.15.172:6789/0 conn(0x7f3b8009ac40 legacy=0x7f3b800b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 monclient: found mon.a 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b801d7b70 con 0x7f3b800ad6f0 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b86bed640 5 monclient: authenticate success, global_id 4122 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b86bed640 10 monclient: _renew_subs 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b86bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b86bed640 1 -- 172.21.15.172:0/4016492014 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b80137470 con 0x7f3b800ad6f0 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b86bed640 10 client.4122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:40.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b86bed640 10 monclient: _renew_subs 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b86bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b86bed640 1 -- 172.21.15.172:0/4016492014 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b80137810 con 0x7f3b800ad6f0 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b86bed640 1 librados: init done 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b78002060 con 0x7f3b800ad6f0 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f3b780015c0 con 0x7f3b800ad6f0 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7effd640 4 set_mon_vals no callback set 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 monclient: dump: 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.938 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3b78005220 con 0x7f3b800ad6f0 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 820819055 0 0) 0x7f3b78006820 con 0x7f3b800ad6f0 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 10 client.4122.objecter ms_dispatch 0x7f3b80000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 3 client.4122.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 3 client.4122.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.943+0000 7f3b7d7fa640 20 client.4122.objecter dump_active .. 0 homeless 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.944+0000 7f3b86bed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.944+0000 7f3b86bed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.944+0000 7f3b86bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.944+0000 7f3b86bed640 1 -- 172.21.15.172:0/4016492014 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b60005080 con 0x7f3b800ad6f0 2024-03-29T11:19:40.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.946+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3b78006aa0 con 0x7f3b800ad6f0 2024-03-29T11:19:40.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.946+0000 7f3b7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:40.946+0000 7f3b7d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:40.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.018+0000 7f3b86bed640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a3fb41ae-75c4-43a8-ae52-b40aa5699776", "id": 1}] 2024-03-29T11:19:40.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.018+0000 7f3b86bed640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a3fb41ae-75c4-43a8-ae52-b40aa5699776", "id": 1}] 2024-03-29T11:19:40.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.018+0000 7f3b86bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:40.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.018+0000 7f3b86bed640 1 -- 172.21.15.172:0/4016492014 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a3fb41ae-75c4-43a8-ae52-b40aa5699776", "id": 1} v 0) v1 -- 0x7f3b600058b0 con 0x7f3b800ad6f0 2024-03-29T11:19:40.997 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.020+0000 7f3b7d7fa640 1 -- 172.21.15.172:0/4016492014 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a3fb41ae-75c4-43a8-ae52-b40aa5699776", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1638440945 0 940483276) 0x7f3b78005880 con 0x7f3b800ad6f0 2024-03-29T11:19:40.997 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.020+0000 7f3b7d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a3fb41ae-75c4-43a8-ae52-b40aa5699776", "id": 1}] 2024-03-29T11:19:40.997 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.020+0000 7f3b7d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:40.997 INFO:teuthology.orchestra.run.smithi172.stdout:1 2024-03-29T11:19:40.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 10 librados: watch_flush enter 2024-03-29T11:19:40.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 10 librados: watch_flush exit 2024-03-29T11:19:40.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 20 client.4122.objecter shutdown clearing up homeless session... 2024-03-29T11:19:40.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 10 client.4122.objecter successfully canceled tick 2024-03-29T11:19:40.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 10 monclient: shutdown 2024-03-29T11:19:40.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:40.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 1 -- 172.21.15.172:0/4016492014 >> v1:172.21.15.59:6789/0 conn(0x7f3b800ad6f0 legacy=0x7f3b80136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 1 -- 172.21.15.172:0/4016492014 shutdown_connections 2024-03-29T11:19:40.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 1 -- 172.21.15.172:0/4016492014 >> 172.21.15.172:0/4016492014 conn(0x7f3b80066b00 msgr2=0x7f3b8013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:40.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 1 -- 172.21.15.172:0/4016492014 shutdown_connections 2024-03-29T11:19:40.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 1 -- 172.21.15.172:0/4016492014 wait complete. 2024-03-29T11:19:41.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.021+0000 7f3b86bed640 1 librados: shutdown 2024-03-29T11:19:41.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 9314facb-46cf-4601-b836-0471cf393809 2 2024-03-29T11:19:41.108 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:41.131+0000 7f1c8c80f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:41.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.131+0000 7fe0044d6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:41.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.131+0000 7fe0044d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.131+0000 7fe0044d6640 10 monclient: monmap: 2024-03-29T11:19:41.109 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:41.109 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.109 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:41.133124+0000 2024-03-29T11:19:41.109 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:41.133124+0000 2024-03-29T11:19:41.109 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.109 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.109 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:41.109 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:41.109 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:41.109 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.132+0000 7fe0044d6640 1 Processor -- start 2024-03-29T11:19:41.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.132+0000 7fe0044d6640 1 -- start start 2024-03-29T11:19:41.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.132+0000 7fe0044d6640 10 monclient: init 2024-03-29T11:19:41.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.132+0000 7fe0044d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.132+0000 7fe0044d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:41.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.132+0000 7fe0044d6640 10 monclient(hunting): picked mon.noname-a con 0x7fdffc066b90 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:41.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 10 monclient(hunting): picked mon.noname-b con 0x7fdffc13f340 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:41.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 10 monclient(hunting): picked mon.noname-c con 0x7fdffc13f7a0 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:41.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdffc13fc30 con 0x7fdffc066b90 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdffc0afd70 con 0x7fdffc13f7a0 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdffc0aec00 con 0x7fdffc13f340 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 10 monclient(hunting): authenticate will time out at 843.392259s 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe002a73640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fdffc13f7a0 0x7fdffc142770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44674/0 (socket says 172.21.15.172:44674) 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe002a73640 1 -- 172.21.15.172:0/947899829 learned_addr learned my addr 172.21.15.172:0/947899829 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752976646 0 0) 0x7fdffc0afd70 con 0x7fdffc13f7a0 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe8003520 con 0x7fdffc13f7a0 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367807196 0 0) 0x7fdffc13fc30 con 0x7fdffc066b90 2024-03-29T11:19:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdffc0afd70 con 0x7fdffc066b90 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917951126 0 0) 0x7fdffc0aec00 con 0x7fdffc13f340 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdffc13fc30 con 0x7fdffc13f340 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1763632023 0 0) 0x7fdfe8003520 con 0x7fdffc13f7a0 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 >> v1:172.21.15.59:6790/0 conn(0x7fdffc13f340 legacy=0x7fdffc142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 >> v1:172.21.15.59:6789/0 conn(0x7fdffc066b90 legacy=0x7fdffc0af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 monclient: found mon.noname-c 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc0aed90 con 0x7fdffc13f7a0 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fdff4001a10 con 0x7fdffc13f7a0 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient: dump: 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdff4001bf0 con 0x7fdffc13f7a0 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe0044d6640 4 set_mon_vals no callback set 2024-03-29T11:19:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 1 -- 172.21.15.172:0/947899829 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fdff4002000 con 0x7fdffc13f7a0 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.133+0000 7fe001270640 10 monclient: dump: 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.134+0000 7fe0044d6640 10 monclient: shutdown 2024-03-29T11:19:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.134+0000 7fe0044d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.134+0000 7fe0044d6640 1 -- 172.21.15.172:0/947899829 >> v1:172.21.15.172:6789/0 conn(0x7fdffc13f7a0 legacy=0x7fdffc142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.134+0000 7fe0044d6640 1 -- 172.21.15.172:0/947899829 shutdown_connections 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.134+0000 7fe0044d6640 1 -- 172.21.15.172:0/947899829 >> 172.21.15.172:0/947899829 conn(0x7fdffc0b4970 msgr2=0x7fdffc0b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.134+0000 7fe0044d6640 1 -- 172.21.15.172:0/947899829 shutdown_connections 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.134+0000 7fe0044d6640 1 -- 172.21.15.172:0/947899829 wait complete. 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 10 monclient: monmap: 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 0.000000 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:created 0.000000 2024-03-29T11:19:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 1 librados: starting msgr at 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 1 librados: starting objecter 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 1 Processor -- start 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 1 -- start start 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 1 librados: setting wanted keys 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 1 librados: calling monclient init 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 10 monclient: init 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 10 monclient(hunting): picked mon.noname-a con 0x7fdffc066b90 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:41.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 10 monclient(hunting): picked mon.noname-b con 0x7fdffc13f340 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 10 monclient(hunting): picked mon.noname-c con 0x7fdffc13f7a0 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdffc138710 con 0x7fdffc066b90 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.135+0000 7fe0044d6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdffc138980 con 0x7fdffc13f340 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fe0044d6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdffc138c60 con 0x7fdffc13f7a0 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fe0044d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fe0044d6640 10 monclient(hunting): authenticate will time out at 843.395192s 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fe001a71640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fdffc13f340 0x7fdffc1411d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44690/0 (socket says 172.21.15.172:44690) 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fe001a71640 1 -- 172.21.15.172:0/1883971763 learned_addr learned my addr 172.21.15.172:0/1883971763 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 client.?.objecter ms_handle_connect 0x7fdffc13f340 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 client.?.objecter ms_handle_connect 0x7fdffc066b90 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 client.?.objecter ms_handle_connect 0x7fdffc13f7a0 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636522861 0 0) 0x7fdffc138980 con 0x7fdffc13f340 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe0004eb0 con 0x7fdffc13f340 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909796071 0 0) 0x7fdffc138710 con 0x7fdffc066b90 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdffc138980 con 0x7fdffc066b90 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441066393 0 0) 0x7fdffc138c60 con 0x7fdffc13f7a0 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdffc138710 con 0x7fdffc13f7a0 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2214858248 0 0) 0x7fdffc138980 con 0x7fdffc066b90 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdffc138c60 con 0x7fdffc066b90 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fdff80017f0 con 0x7fdffc066b90 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3293049942 0 0) 0x7fdfe0004eb0 con 0x7fdffc13f340 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdffc138980 con 0x7fdffc13f340 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3177703521 0 0) 0x7fdffc138710 con 0x7fdffc13f7a0 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfe0004eb0 con 0x7fdffc13f7a0 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fdfec001990 con 0x7fdffc13f340 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fdff4001d60 con 0x7fdffc13f7a0 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.131 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2918973747 0 0) 0x7fdffc138c60 con 0x7fdffc066b90 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 >> v1:172.21.15.59:6790/0 conn(0x7fdffc13f7a0 legacy=0x7fdffc1e1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 >> v1:172.21.15.172:6789/0 conn(0x7fdffc13f340 legacy=0x7fdffc1411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 monclient: found mon.a 2024-03-29T11:19:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc137950 con 0x7fdffc066b90 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fdff2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fe0044d6640 5 monclient: authenticate success, global_id 4128 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fe0044d6640 10 monclient: _renew_subs 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fe0044d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.136+0000 7fe0044d6640 1 -- 172.21.15.172:0/1883971763 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdffc136600 con 0x7fdffc066b90 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fe0044d6640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fe0044d6640 10 monclient: _renew_subs 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fe0044d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fe0044d6640 1 -- 172.21.15.172:0/1883971763 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdffc13a4b0 con 0x7fdffc066b90 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fe0044d6640 1 librados: init done 2024-03-29T11:19:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdff80035f0 con 0x7fdffc066b90 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fdff8003900 con 0x7fdffc066b90 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fe000a6f640 4 set_mon_vals no callback set 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 10 monclient: dump: 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdff8003df0 con 0x7fdffc066b90 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 12798832 0 0) 0x7fdff8005360 con 0x7fdffc066b90 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 10 client.4128.objecter ms_dispatch 0x7fdffc000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 3 client.4128.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 3 client.4128.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fdff2ffd640 20 client.4128.objecter dump_active .. 0 homeless 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fe0044d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fe0044d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fe0044d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.137+0000 7fe0044d6640 1 -- 172.21.15.172:0/1883971763 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe4005080 con 0x7fdffc066b90 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.139+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdff8005050 con 0x7fdffc066b90 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.140+0000 7fdff2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.140+0000 7fdff2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:41.136 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:41.149+0000 7fa3c7432040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:41.165 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:41.188+0000 7f1c8c80f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:41.181 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:41.205+0000 7fa3c7432040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:41.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.212+0000 7fe0044d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9314facb-46cf-4601-b836-0471cf393809", "id": 2}] 2024-03-29T11:19:41.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.212+0000 7fe0044d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9314facb-46cf-4601-b836-0471cf393809", "id": 2}] 2024-03-29T11:19:41.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.212+0000 7fe0044d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:41.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.212+0000 7fe0044d6640 1 -- 172.21.15.172:0/1883971763 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9314facb-46cf-4601-b836-0471cf393809", "id": 2} v 0) v1 -- 0x7fdfe40058b0 con 0x7fdffc066b90 2024-03-29T11:19:41.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fdff2ffd640 1 -- 172.21.15.172:0/1883971763 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9314facb-46cf-4601-b836-0471cf393809", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1612632849 0 216599125) 0x7fdff80041e0 con 0x7fdffc066b90 2024-03-29T11:19:41.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fdff2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9314facb-46cf-4601-b836-0471cf393809", "id": 2}] 2024-03-29T11:19:41.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fdff2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:41.191 INFO:teuthology.orchestra.run.smithi172.stdout:2 2024-03-29T11:19:41.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fe0044d6640 10 librados: watch_flush enter 2024-03-29T11:19:41.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fe0044d6640 10 librados: watch_flush exit 2024-03-29T11:19:41.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fe0044d6640 20 client.4128.objecter shutdown clearing up homeless session... 2024-03-29T11:19:41.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fe0044d6640 10 client.4128.objecter successfully canceled tick 2024-03-29T11:19:41.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fe0044d6640 10 monclient: shutdown 2024-03-29T11:19:41.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fe0044d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fe0044d6640 1 -- 172.21.15.172:0/1883971763 >> v1:172.21.15.59:6789/0 conn(0x7fdffc066b90 legacy=0x7fdffc1371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fe0044d6640 1 -- 172.21.15.172:0/1883971763 shutdown_connections 2024-03-29T11:19:41.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fe0044d6640 1 -- 172.21.15.172:0/1883971763 >> 172.21.15.172:0/1883971763 conn(0x7fdffc0b4970 msgr2=0x7fdffc0b3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fe0044d6640 1 -- 172.21.15.172:0/1883971763 shutdown_connections 2024-03-29T11:19:41.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.214+0000 7fe0044d6640 1 -- 172.21.15.172:0/1883971763 wait complete. 2024-03-29T11:19:41.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.215+0000 7fe0044d6640 1 librados: shutdown 2024-03-29T11:19:41.202 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 27bf268d-5153-49bb-af84-46d68b59c5e0 3 2024-03-29T11:19:41.213 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:41.236+0000 7f1c8c80f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:41.229 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:41.252+0000 7fa3c7432040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:41.270 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:41.293+0000 7f1c8c80f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:41.285 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:41.308+0000 7fa3c7432040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:41.302 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.325+0000 7fa40d24e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.325+0000 7fa40d24e640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.325+0000 7fa40d24e640 10 monclient: monmap: 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:41.326660+0000 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:41.326660+0000 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.325+0000 7fa40d24e640 1 Processor -- start 2024-03-29T11:19:41.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.325+0000 7fa40d24e640 1 -- start start 2024-03-29T11:19:41.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.325+0000 7fa40d24e640 10 monclient: init 2024-03-29T11:19:41.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40d24e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40d24e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:41.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40d24e640 10 monclient(hunting): picked mon.noname-b con 0x7fa40809ac40 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:41.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40d24e640 10 monclient(hunting): picked mon.noname-c con 0x7fa40809c870 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:41.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40d24e640 10 monclient(hunting): picked mon.noname-a con 0x7fa4080b4360 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:41.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40d24e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40809dec0 con 0x7fa4080b4360 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40d24e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40809da30 con 0x7fa40809c870 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40d24e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4080b39c0 con 0x7fa40809ac40 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40d24e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40d24e640 10 monclient(hunting): authenticate will time out at 843.585655s 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40659c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa40809c870 0x7fa40813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44702/0 (socket says 172.21.15.172:44702) 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa40659c640 1 -- 172.21.15.172:0/3011972745 learned_addr learned my addr 172.21.15.172:0/3011972745 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167862249 0 0) 0x7fa40809da30 con 0x7fa40809c870 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f0003520 con 0x7fa40809c870 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492566311 0 0) 0x7fa4080b39c0 con 0x7fa40809ac40 2024-03-29T11:19:41.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa40809da30 con 0x7fa40809ac40 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202638531 0 0) 0x7fa400001490 con 0x7fa4080b4360 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.326+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4080b39c0 con 0x7fa4080b4360 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2675494850 0 0) 0x7fa3f0003520 con 0x7fa40809c870 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 >> v1:172.21.15.59:6790/0 conn(0x7fa40809ac40 legacy=0x7fa4080b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 >> v1:172.21.15.59:6789/0 conn(0x7fa4080b4360 legacy=0x7fa40814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 1 monclient: found mon.noname-c 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4080b5160 con 0x7fa40809c870 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa3f8001690 con 0x7fa40809c870 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 10 monclient: dump: 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.315 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa40d24e640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa40d24e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3f8001870 con 0x7fa40809c870 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 1 -- 172.21.15.172:0/3011972745 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa3f8001c80 con 0x7fa40809c870 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:41.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa405d9b640 10 monclient: dump: 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa40d24e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa40d24e640 4 set_mon_vals no callback set 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa40d24e640 10 monclient: shutdown 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa40d24e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa40d24e640 1 -- 172.21.15.172:0/3011972745 >> v1:172.21.15.172:6789/0 conn(0x7fa40809c870 legacy=0x7fa40813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa40d24e640 1 -- 172.21.15.172:0/3011972745 shutdown_connections 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa40d24e640 1 -- 172.21.15.172:0/3011972745 >> 172.21.15.172:0/3011972745 conn(0x7fa408066b00 msgr2=0x7fa408066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa40d24e640 1 -- 172.21.15.172:0/3011972745 shutdown_connections 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.327+0000 7fa40d24e640 1 -- 172.21.15.172:0/3011972745 wait complete. 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.328+0000 7fa40d24e640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.328+0000 7fa40d24e640 10 monclient: monmap: 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 0.000000 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:created 0.000000 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:41.318 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.328+0000 7fa40d24e640 1 librados: starting msgr at 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.328+0000 7fa40d24e640 1 librados: starting objecter 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 1 Processor -- start 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 1 -- start start 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 1 librados: setting wanted keys 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 1 librados: calling monclient init 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 10 monclient: init 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 10 monclient(hunting): picked mon.noname-a con 0x7fa40809ac40 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 10 monclient(hunting): picked mon.noname-b con 0x7fa4080af970 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 10 monclient(hunting): picked mon.noname-c con 0x7fa408139360 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4081d7200 con 0x7fa40809ac40 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4080b1a80 con 0x7fa4080af970 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4080acd70 con 0x7fa408139360 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40d24e640 10 monclient(hunting): authenticate will time out at 843.588788s 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40659c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa4080af970 0x7fa4081d7690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44704/0 (socket says 172.21.15.172:44704) 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa40659c640 1 -- 172.21.15.172:0/2494447623 learned_addr learned my addr 172.21.15.172:0/2494447623 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 10 client.?.objecter ms_handle_connect 0x7fa4080af970 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 10 client.?.objecter ms_handle_connect 0x7fa408139360 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 10 client.?.objecter ms_handle_connect 0x7fa40809ac40 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198440840 0 0) 0x7fa4080b1a80 con 0x7fa4080af970 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.329+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f4004eb0 con 0x7fa4080af970 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558472416 0 0) 0x7fa4080acd70 con 0x7fa408139360 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4080b1a80 con 0x7fa408139360 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839867972 0 0) 0x7fa4081d7200 con 0x7fa40809ac40 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4080acd70 con 0x7fa40809ac40 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3623436334 0 0) 0x7fa3f4004eb0 con 0x7fa4080af970 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4081d7200 con 0x7fa4080af970 2024-03-29T11:19:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa3f8001bc0 con 0x7fa4080af970 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2451825108 0 0) 0x7fa4080b1a80 con 0x7fa408139360 2024-03-29T11:19:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3f4004eb0 con 0x7fa408139360 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa400001b40 con 0x7fa408139360 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2387297925 0 0) 0x7fa4080acd70 con 0x7fa40809ac40 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4080b1a80 con 0x7fa40809ac40 2024-03-29T11:19:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa3fc0018b0 con 0x7fa40809ac40 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2379816697 0 0) 0x7fa4081d7200 con 0x7fa4080af970 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 >> v1:172.21.15.59:6790/0 conn(0x7fa408139360 legacy=0x7fa4081397b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 >> v1:172.21.15.59:6789/0 conn(0x7fa40809ac40 legacy=0x7fa4080b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 monclient: found mon.b 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4080acf00 con 0x7fa4080af970 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa40d24e640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa40d24e640 10 monclient: _renew_subs 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa40d24e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa40d24e640 1 -- 172.21.15.172:0/2494447623 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4081d7da0 con 0x7fa4080af970 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa40d24e640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa40d24e640 10 monclient: _renew_subs 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa40d24e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa40d24e640 1 -- 172.21.15.172:0/2494447623 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4081e3b50 con 0x7fa4080af970 2024-03-29T11:19:41.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa40d24e640 1 librados: init done 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3f8004bf0 con 0x7fa4080af970 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa3f8004f00 con 0x7fa4080af970 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa40559a640 4 set_mon_vals no callback set 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 monclient: dump: 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.326 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa3f80053f0 con 0x7fa4080af970 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2281838260 0 0) 0x7fa3f8006800 con 0x7fa4080af970 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 10 client.4129.objecter ms_dispatch 0x7fa408000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.330+0000 7fa3ef7fe640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.331+0000 7fa40d24e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.331+0000 7fa40d24e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.331+0000 7fa40d24e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.331+0000 7fa40d24e640 1 -- 172.21.15.172:0/2494447623 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e8005080 con 0x7fa4080af970 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.333+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa3f8006480 con 0x7fa4080af970 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.333+0000 7fa3ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.333+0000 7fa3ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:41.328 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:41.339+0000 7f1c8c80f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:41.329 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:41.352+0000 7fa3c7432040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:41.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.406+0000 7fa40d24e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "27bf268d-5153-49bb-af84-46d68b59c5e0", "id": 3}] 2024-03-29T11:19:41.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.406+0000 7fa40d24e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "27bf268d-5153-49bb-af84-46d68b59c5e0", "id": 3}] 2024-03-29T11:19:41.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.406+0000 7fa40d24e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.406+0000 7fa40d24e640 1 -- 172.21.15.172:0/2494447623 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd new", "uuid": "27bf268d-5153-49bb-af84-46d68b59c5e0", "id": 3} v 0) v1 -- 0x7fa3e80058b0 con 0x7fa4080af970 2024-03-29T11:19:41.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.410+0000 7fa3ef7fe640 1 -- 172.21.15.172:0/2494447623 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "27bf268d-5153-49bb-af84-46d68b59c5e0", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3146063708 0 525046306) 0x7fa3f80069c0 con 0x7fa4080af970 2024-03-29T11:19:41.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.410+0000 7fa3ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "27bf268d-5153-49bb-af84-46d68b59c5e0", "id": 3}] 2024-03-29T11:19:41.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.410+0000 7fa3ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:41.387 INFO:teuthology.orchestra.run.smithi172.stdout:3 2024-03-29T11:19:41.388 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 10 librados: watch_flush enter 2024-03-29T11:19:41.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 10 librados: watch_flush exit 2024-03-29T11:19:41.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:19:41.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:19:41.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 10 monclient: shutdown 2024-03-29T11:19:41.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 1 -- 172.21.15.172:0/2494447623 >> v1:172.21.15.172:6789/0 conn(0x7fa4080af970 legacy=0x7fa4081d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 1 -- 172.21.15.172:0/2494447623 shutdown_connections 2024-03-29T11:19:41.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 1 -- 172.21.15.172:0/2494447623 >> 172.21.15.172:0/2494447623 conn(0x7fa408066b00 msgr2=0x7fa40813b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 1 -- 172.21.15.172:0/2494447623 shutdown_connections 2024-03-29T11:19:41.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 1 -- 172.21.15.172:0/2494447623 wait complete. 2024-03-29T11:19:41.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.411+0000 7fa40d24e640 1 librados: shutdown 2024-03-29T11:19:41.398 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new a26fa133-b0ec-42d4-b9ba-d6150e5b456b 4 2024-03-29T11:19:41.437 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:41.460+0000 7f1c8c80f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:41.449 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:41.473+0000 7fa3c7432040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:41.511 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:41.534+0000 7f1c8c80f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:41.522 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:41.546+0000 7fa3c7432040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.560+0000 7f720978f640 10 monclient: get_monmap_and_config 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.560+0000 7f720978f640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.560+0000 7f720978f640 10 monclient: monmap: 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:41.562432+0000 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:41.562432+0000 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:41.538 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:41.539 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.561+0000 7f720978f640 1 Processor -- start 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.561+0000 7f720978f640 1 -- start start 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.561+0000 7f720978f640 10 monclient: init 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f720978f640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f720978f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f720978f640 10 monclient(hunting): picked mon.noname-a con 0x7f720409ac40 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f720978f640 10 monclient(hunting): picked mon.noname-c con 0x7f720409c870 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f720978f640 10 monclient(hunting): picked mon.noname-b con 0x7f72040b4360 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f720978f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720409dec0 con 0x7f720409ac40 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f720978f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720409da30 con 0x7f720409c870 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f720978f640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72040b39c0 con 0x7f72040b4360 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f720978f640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f720978f640 10 monclient(hunting): authenticate will time out at 843.821504s 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f71fbfff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f720409c870 0x7f720413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44714/0 (socket says 172.21.15.172:44714) 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f71fbfff640 1 -- 172.21.15.172:0/2892644828 learned_addr learned my addr 172.21.15.172:0/2892644828 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039722674 0 0) 0x7f720409da30 con 0x7f720409c870 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f0003520 con 0x7f720409c870 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761566569 0 0) 0x7f72040b39c0 con 0x7f72040b4360 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f720409da30 con 0x7f72040b4360 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852118265 0 0) 0x7f71f4001110 con 0x7f720409ac40 2024-03-29T11:19:41.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72040b39c0 con 0x7f720409ac40 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1877836672 0 0) 0x7f71f0003520 con 0x7f720409c870 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 >> v1:172.21.15.59:6790/0 conn(0x7f72040b4360 legacy=0x7f720414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 >> v1:172.21.15.59:6789/0 conn(0x7f720409ac40 legacy=0x7f72040b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 1 monclient: found mon.noname-c 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.562+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72040b5160 con 0x7f720409c870 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:19:41.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f71ec001690 con 0x7f720409c870 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 10 monclient: dump: 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:41.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71ec0018e0 con 0x7f720409c870 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 1 -- 172.21.15.172:0/2892644828 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f71ec001c70 con 0x7f720409c870 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 4 set_mon_vals no callback set 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f72027fc640 10 monclient: dump: 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.550 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 10 monclient: shutdown 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 1 -- 172.21.15.172:0/2892644828 >> v1:172.21.15.172:6789/0 conn(0x7f720409c870 legacy=0x7f720413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 1 -- 172.21.15.172:0/2892644828 shutdown_connections 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 1 -- 172.21.15.172:0/2892644828 >> 172.21.15.172:0/2892644828 conn(0x7f7204066b00 msgr2=0x7f7204066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 1 -- 172.21.15.172:0/2892644828 shutdown_connections 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.563+0000 7f720978f640 1 -- 172.21.15.172:0/2892644828 wait complete. 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.564+0000 7f720978f640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.564+0000 7f720978f640 10 monclient: monmap: 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 0.000000 2024-03-29T11:19:41.551 INFO:teuthology.orchestra.run.smithi172.stderr:created 0.000000 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.564+0000 7f720978f640 1 librados: starting msgr at 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.564+0000 7f720978f640 1 librados: starting objecter 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.564+0000 7f720978f640 1 Processor -- start 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.564+0000 7f720978f640 1 -- start start 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.564+0000 7f720978f640 1 librados: setting wanted keys 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.564+0000 7f720978f640 1 librados: calling monclient init 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.564+0000 7f720978f640 10 monclient: init 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f720978f640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f720978f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f720978f640 10 monclient(hunting): picked mon.noname-c con 0x7f720409ac40 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f720978f640 10 monclient(hunting): picked mon.noname-b con 0x7f7204139250 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f720978f640 10 monclient(hunting): picked mon.noname-a con 0x7f72040b02f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f720978f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72040b1b40 con 0x7f72040b02f0 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f720978f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72041d83a0 con 0x7f7204139250 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f720978f640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72040b0760 con 0x7f720409ac40 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f720978f640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f720978f640 10 monclient(hunting): authenticate will time out at 843.824453s 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fbfff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7204139250 0x7f72040afbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44724/0 (socket says 172.21.15.172:44724) 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fbfff640 1 -- 172.21.15.172:0/4177023050 learned_addr learned my addr 172.21.15.172:0/4177023050 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 client.?.objecter ms_handle_connect 0x7f7204139250 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 client.?.objecter ms_handle_connect 0x7f720409ac40 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 client.?.objecter ms_handle_connect 0x7f72040b02f0 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062511652 0 0) 0x7f72041d83a0 con 0x7f7204139250 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71e4004eb0 con 0x7f7204139250 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511260700 0 0) 0x7f72040b0760 con 0x7f720409ac40 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72041d83a0 con 0x7f720409ac40 2024-03-29T11:19:41.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343509517 0 0) 0x7f72040b1b40 con 0x7f72040b02f0 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72040b0760 con 0x7f72040b02f0 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 62386168 0 0) 0x7f72041d83a0 con 0x7f720409ac40 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72040b1b40 con 0x7f720409ac40 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1649614678 0 0) 0x7f71e4004eb0 con 0x7f7204139250 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72041d83a0 con 0x7f7204139250 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f71f4001760 con 0x7f720409ac40 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f71ec001bf0 con 0x7f7204139250 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 34736475 0 0) 0x7f72040b0760 con 0x7f72040b02f0 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71e4004eb0 con 0x7f72040b02f0 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f71fc001b50 con 0x7f72040b02f0 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.565+0000 7f71fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.557 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3224348472 0 0) 0x7f72040b1b40 con 0x7f720409ac40 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 >> v1:172.21.15.172:6789/0 conn(0x7f7204139250 legacy=0x7f72040afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 >> v1:172.21.15.59:6789/0 conn(0x7f72040b02f0 legacy=0x7f72041e20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 1 monclient: found mon.c 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72040b08f0 con 0x7f720409ac40 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f720978f640 5 monclient: authenticate success, global_id 4130 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f720978f640 10 monclient: _renew_subs 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f720978f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f720978f640 1 -- 172.21.15.172:0/4177023050 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72041d7fa0 con 0x7f720409ac40 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f720978f640 10 client.4130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f720978f640 10 monclient: _renew_subs 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f720978f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f720978f640 1 -- 172.21.15.172:0/4177023050 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72041e39e0 con 0x7f720409ac40 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f720978f640 1 librados: init done 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71f4001c30 con 0x7f720409ac40 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f71f4003610 con 0x7f720409ac40 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f7201ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 10 monclient: dump: 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f71f4003b00 con 0x7f720409ac40 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1414375337 0 0) 0x7f71f40042c0 con 0x7f720409ac40 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 10 client.4130.objecter ms_dispatch 0x7f7204000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 3 client.4130.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 3 client.4130.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f71fb7fe640 20 client.4130.objecter dump_active .. 0 homeless 2024-03-29T11:19:41.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f720978f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.566+0000 7f720978f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.567+0000 7f720978f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:19:41.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.567+0000 7f720978f640 1 -- 172.21.15.172:0/4177023050 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e8005080 con 0x7f720409ac40 2024-03-29T11:19:41.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.569+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f71f4004510 con 0x7f720409ac40 2024-03-29T11:19:41.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.569+0000 7f71fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.569+0000 7f71fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:41.561 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:41.584+0000 7f1c8c80f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:41.570 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:41.594+0000 7fa3c7432040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:41.615 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:41.638+0000 7f1c8c80f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:41.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.641+0000 7f720978f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a26fa133-b0ec-42d4-b9ba-d6150e5b456b", "id": 4}] 2024-03-29T11:19:41.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.641+0000 7f720978f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a26fa133-b0ec-42d4-b9ba-d6150e5b456b", "id": 4}] 2024-03-29T11:19:41.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.641+0000 7f720978f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:19:41.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.641+0000 7f720978f640 1 -- 172.21.15.172:0/4177023050 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd new", "uuid": "a26fa133-b0ec-42d4-b9ba-d6150e5b456b", "id": 4} v 0) v1 -- 0x7f71e80058b0 con 0x7f720409ac40 2024-03-29T11:19:41.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.644+0000 7f71fb7fe640 1 -- 172.21.15.172:0/4177023050 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a26fa133-b0ec-42d4-b9ba-d6150e5b456b", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2500169590 0 1697012583) 0x7f71f4003db0 con 0x7f720409ac40 2024-03-29T11:19:41.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.644+0000 7f71fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a26fa133-b0ec-42d4-b9ba-d6150e5b456b", "id": 4}] 2024-03-29T11:19:41.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.644+0000 7f71fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:41.621 INFO:teuthology.orchestra.run.smithi172.stdout:4 2024-03-29T11:19:41.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 10 librados: watch_flush enter 2024-03-29T11:19:41.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 10 librados: watch_flush exit 2024-03-29T11:19:41.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 20 client.4130.objecter shutdown clearing up homeless session... 2024-03-29T11:19:41.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 10 client.4130.objecter successfully canceled tick 2024-03-29T11:19:41.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 10 monclient: shutdown 2024-03-29T11:19:41.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 1 -- 172.21.15.172:0/4177023050 >> v1:172.21.15.59:6790/0 conn(0x7f720409ac40 legacy=0x7f72041d7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 1 -- 172.21.15.172:0/4177023050 shutdown_connections 2024-03-29T11:19:41.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 1 -- 172.21.15.172:0/4177023050 >> 172.21.15.172:0/4177023050 conn(0x7f7204066b00 msgr2=0x7f720413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 1 -- 172.21.15.172:0/4177023050 shutdown_connections 2024-03-29T11:19:41.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 1 -- 172.21.15.172:0/4177023050 wait complete. 2024-03-29T11:19:41.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.645+0000 7f720978f640 1 librados: shutdown 2024-03-29T11:19:41.625 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:41.648+0000 7fa3c7432040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:41.632 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new c907b48d-1121-4419-9221-28db3efd1054 5 2024-03-29T11:19:41.666 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:41.689+0000 7f1c8c80f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:41.674 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:41.697+0000 7fa3c7432040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:41.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.754+0000 7f36549e6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:41.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.754+0000 7f36549e6640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.754+0000 7f36549e6640 10 monclient: monmap: 2024-03-29T11:19:41.732 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:41.732 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.732 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:41.756374+0000 2024-03-29T11:19:41.732 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:41.756374+0000 2024-03-29T11:19:41.732 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.732 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.732 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:41.732 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:41.733 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:41.733 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.755+0000 7f36549e6640 1 Processor -- start 2024-03-29T11:19:41.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.755+0000 7f36549e6640 1 -- start start 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.755+0000 7f36549e6640 10 monclient: init 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 10 monclient(hunting): picked mon.noname-b con 0x7f365009ac40 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 10 monclient(hunting): picked mon.noname-a con 0x7f365009c870 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 10 monclient(hunting): picked mon.noname-c con 0x7f36500b4360 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365009dec0 con 0x7f365009c870 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365009da30 con 0x7f36500b4360 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36500b39c0 con 0x7f365009ac40 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 10 monclient(hunting): authenticate will time out at 844.015533s 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364ed9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f36500b4360 0x7f365014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44734/0 (socket says 172.21.15.172:44734) 2024-03-29T11:19:41.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364ed9d640 1 -- 172.21.15.172:0/2748731846 learned_addr learned my addr 172.21.15.172:0/2748731846 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716220261 0 0) 0x7f365009da30 con 0x7f36500b4360 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f363c003520 con 0x7f36500b4360 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147015431 0 0) 0x7f36500b39c0 con 0x7f365009ac40 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f365009da30 con 0x7f365009ac40 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954226076 0 0) 0x7f3638001110 con 0x7f365009c870 2024-03-29T11:19:41.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36500b39c0 con 0x7f365009c870 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1375550567 0 0) 0x7f363c003520 con 0x7f36500b4360 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 >> v1:172.21.15.59:6790/0 conn(0x7f365009ac40 legacy=0x7f36500b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 >> v1:172.21.15.59:6789/0 conn(0x7f365009c870 legacy=0x7f365013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 monclient: found mon.noname-c 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36500b5160 con 0x7f36500b4360 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f3648001a10 con 0x7f36500b4360 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f364dd9b640 10 monclient: dump: 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.756+0000 7f36549e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3648001b70 con 0x7f36500b4360 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f364dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f364dd9b640 1 -- 172.21.15.172:0/2748731846 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f3648001f80 con 0x7f36500b4360 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f36549e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f36549e6640 4 set_mon_vals no callback set 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f364dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f364dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f364dd9b640 10 monclient: dump: 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.746 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f36549e6640 10 monclient: shutdown 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f36549e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f36549e6640 1 -- 172.21.15.172:0/2748731846 >> v1:172.21.15.172:6789/0 conn(0x7f36500b4360 legacy=0x7f365014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f36549e6640 1 -- 172.21.15.172:0/2748731846 shutdown_connections 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f36549e6640 1 -- 172.21.15.172:0/2748731846 >> 172.21.15.172:0/2748731846 conn(0x7f3650066b00 msgr2=0x7f3650066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f36549e6640 1 -- 172.21.15.172:0/2748731846 shutdown_connections 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.757+0000 7f36549e6640 1 -- 172.21.15.172:0/2748731846 wait complete. 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.758+0000 7f36549e6640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.758+0000 7f36549e6640 10 monclient: monmap: 2024-03-29T11:19:41.747 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 0.000000 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:created 0.000000 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.758+0000 7f36549e6640 1 librados: starting msgr at 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.758+0000 7f36549e6640 1 librados: starting objecter 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.758+0000 7f36549e6640 1 Processor -- start 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.758+0000 7f36549e6640 1 -- start start 2024-03-29T11:19:41.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.758+0000 7f36549e6640 1 librados: setting wanted keys 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.758+0000 7f36549e6640 1 librados: calling monclient init 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.758+0000 7f36549e6640 10 monclient: init 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f36549e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f36549e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f36549e6640 10 monclient(hunting): picked mon.noname-a con 0x7f365009ac40 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f36549e6640 10 monclient(hunting): picked mon.noname-b con 0x7f36500aef20 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f36549e6640 10 monclient(hunting): picked mon.noname-c con 0x7f36500ae8e0 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f36549e6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36500afec0 con 0x7f365009ac40 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f36549e6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36500adb40 con 0x7f36500aef20 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f36549e6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36501d7a20 con 0x7f36500ae8e0 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f36549e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f36549e6640 10 monclient(hunting): authenticate will time out at 844.018514s 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3647fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f36500aef20 0x7f36501d7f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44750/0 (socket says 172.21.15.172:44750) 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3647fff640 1 -- 172.21.15.172:0/3845066521 learned_addr learned my addr 172.21.15.172:0/3845066521 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 client.?.objecter ms_handle_connect 0x7f36500aef20 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 client.?.objecter ms_handle_connect 0x7f365009ac40 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 client.?.objecter ms_handle_connect 0x7f36500ae8e0 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265242697 0 0) 0x7f36500adb40 con 0x7f36500aef20 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3630004eb0 con 0x7f36500aef20 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600935062 0 0) 0x7f36500afec0 con 0x7f365009ac40 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36500adb40 con 0x7f365009ac40 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769680797 0 0) 0x7f36501d7a20 con 0x7f36500ae8e0 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36500afec0 con 0x7f36500ae8e0 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3875497551 0 0) 0x7f3630004eb0 con 0x7f36500aef20 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36501d7a20 con 0x7f36500aef20 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f36380017e0 con 0x7f36500aef20 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:41.751 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3681226588 0 0) 0x7f36500adb40 con 0x7f365009ac40 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3630004eb0 con 0x7f365009ac40 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f3640001970 con 0x7f365009ac40 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.759+0000 7f3646ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3474406688 0 0) 0x7f36500afec0 con 0x7f36500ae8e0 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36500adb40 con 0x7f36500ae8e0 2024-03-29T11:19:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f3648001e80 con 0x7f36500ae8e0 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 489538263 0 0) 0x7f36501d7a20 con 0x7f36500aef20 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 >> v1:172.21.15.59:6790/0 conn(0x7f36500ae8e0 legacy=0x7f36500b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 >> v1:172.21.15.59:6789/0 conn(0x7f365009ac40 legacy=0x7f36500b0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 monclient: found mon.b 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36501d7640 con 0x7f36500aef20 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f36549e6640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f36549e6640 10 monclient: _renew_subs 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f36549e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f36549e6640 1 -- 172.21.15.172:0/3845066521 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36500b1710 con 0x7f36500aef20 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f36549e6640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f36549e6640 10 monclient: _renew_subs 2024-03-29T11:19:41.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f36549e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f36549e6640 1 -- 172.21.15.172:0/3845066521 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36501e3e90 con 0x7f36500aef20 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f36549e6640 1 librados: init done 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3638001cd0 con 0x7f36500aef20 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f36380037b0 con 0x7f36500aef20 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f364d59a640 4 set_mon_vals no callback set 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 monclient: dump: 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.756 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3638003d60 con 0x7f36500aef20 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 840076445 0 0) 0x7f3638006690 con 0x7f36500aef20 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 10 client.4141.objecter ms_dispatch 0x7f3650000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.760+0000 7f3646ffd640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.761+0000 7f36549e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.761+0000 7f36549e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.761+0000 7f36549e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.761+0000 7f36549e6640 1 -- 172.21.15.172:0/3845066521 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3634005080 con 0x7f36500aef20 2024-03-29T11:19:41.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.763+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3638004240 con 0x7f36500aef20 2024-03-29T11:19:41.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.763+0000 7f3646ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.763+0000 7f3646ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:41.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.835+0000 7f36549e6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c907b48d-1121-4419-9221-28db3efd1054", "id": 5}] 2024-03-29T11:19:41.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.835+0000 7f36549e6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c907b48d-1121-4419-9221-28db3efd1054", "id": 5}] 2024-03-29T11:19:41.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.835+0000 7f36549e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.835+0000 7f36549e6640 1 -- 172.21.15.172:0/3845066521 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c907b48d-1121-4419-9221-28db3efd1054", "id": 5} v 0) v1 -- 0x7f36340058b0 con 0x7f36500aef20 2024-03-29T11:19:41.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f3646ffd640 1 -- 172.21.15.172:0/3845066521 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c907b48d-1121-4419-9221-28db3efd1054", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3250768020 0 1988412176) 0x7f3638006110 con 0x7f36500aef20 2024-03-29T11:19:41.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f3646ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c907b48d-1121-4419-9221-28db3efd1054", "id": 5}] 2024-03-29T11:19:41.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f3646ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:41.816 INFO:teuthology.orchestra.run.smithi172.stdout:5 2024-03-29T11:19:41.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f36549e6640 10 librados: watch_flush enter 2024-03-29T11:19:41.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f36549e6640 10 librados: watch_flush exit 2024-03-29T11:19:41.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f36549e6640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:19:41.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f36549e6640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:19:41.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f36549e6640 10 monclient: shutdown 2024-03-29T11:19:41.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f36549e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f36549e6640 1 -- 172.21.15.172:0/3845066521 >> v1:172.21.15.172:6789/0 conn(0x7f36500aef20 legacy=0x7f36501d7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f36549e6640 1 -- 172.21.15.172:0/3845066521 shutdown_connections 2024-03-29T11:19:41.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.839+0000 7f36549e6640 1 -- 172.21.15.172:0/3845066521 >> 172.21.15.172:0/3845066521 conn(0x7f3650066b00 msgr2=0x7f365013a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.840+0000 7f36549e6640 1 -- 172.21.15.172:0/3845066521 shutdown_connections 2024-03-29T11:19:41.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.840+0000 7f36549e6640 1 -- 172.21.15.172:0/3845066521 wait complete. 2024-03-29T11:19:41.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.840+0000 7f36549e6640 1 librados: shutdown 2024-03-29T11:19:41.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3 6 2024-03-29T11:19:41.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.991+0000 7f997926e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.991+0000 7f997926e640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.991+0000 7f997926e640 10 monclient: monmap: 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:41.993065+0000 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:41.993065+0000 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:41.969 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 1 Processor -- start 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 1 -- start start 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 10 monclient: init 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 10 monclient(hunting): picked mon.noname-b con 0x7f99740af360 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 10 monclient(hunting): picked mon.noname-c con 0x7f9974146320 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 10 monclient(hunting): picked mon.noname-a con 0x7f997406b730 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997406c650 con 0x7f997406b730 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997406bb90 con 0x7f9974146320 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99740aff00 con 0x7f99740af360 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f997926e640 10 monclient(hunting): authenticate will time out at 844.252034s 2024-03-29T11:19:41.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f99727fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9974146320 0x7f997406b020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44758/0 (socket says 172.21.15.172:44758) 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.992+0000 7f99727fc640 1 -- 172.21.15.172:0/741317762 learned_addr learned my addr 172.21.15.172:0/741317762 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50144844 0 0) 0x7f997406bb90 con 0x7f9974146320 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f995c003520 con 0x7f9974146320 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443471571 0 0) 0x7f997406c650 con 0x7f997406b730 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997406bb90 con 0x7f997406b730 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030284243 0 0) 0x7f99740aff00 con 0x7f99740af360 2024-03-29T11:19:41.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997406c650 con 0x7f99740af360 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126941137 0 0) 0x7f995c003520 con 0x7f9974146320 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 >> v1:172.21.15.59:6790/0 conn(0x7f99740af360 legacy=0x7f9974145c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 >> v1:172.21.15.59:6789/0 conn(0x7f997406b730 legacy=0x7f997414da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997406cae0 con 0x7f9974146320 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9968001690 con 0x7f9974146320 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f997926e640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f997926e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient: dump: 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.982 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f997926e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9968001870 con 0x7f9974146320 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 1 -- 172.21.15.172:0/741317762 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9968001c80 con 0x7f9974146320 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f997926e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f997926e640 4 set_mon_vals no callback set 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.993+0000 7f9971ffb640 10 monclient: dump: 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.994+0000 7f997926e640 10 monclient: shutdown 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.994+0000 7f997926e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.994+0000 7f997926e640 1 -- 172.21.15.172:0/741317762 >> v1:172.21.15.172:6789/0 conn(0x7f9974146320 legacy=0x7f997406b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.994+0000 7f997926e640 1 -- 172.21.15.172:0/741317762 shutdown_connections 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.994+0000 7f997926e640 1 -- 172.21.15.172:0/741317762 >> 172.21.15.172:0/741317762 conn(0x7f99740b58f0 msgr2=0x7f99740b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.994+0000 7f997926e640 1 -- 172.21.15.172:0/741317762 shutdown_connections 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.994+0000 7f997926e640 1 -- 172.21.15.172:0/741317762 wait complete. 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.994+0000 7f997926e640 10 monclient: build_initial_monmap 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.994+0000 7f997926e640 10 monclient: monmap: 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:41.984 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 0.000000 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:created 0.000000 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.995+0000 7f997926e640 1 librados: starting msgr at 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.995+0000 7f997926e640 1 librados: starting objecter 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.995+0000 7f997926e640 1 Processor -- start 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.995+0000 7f997926e640 1 -- start start 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.995+0000 7f997926e640 1 librados: setting wanted keys 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.995+0000 7f997926e640 1 librados: calling monclient init 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.995+0000 7f997926e640 10 monclient: init 2024-03-29T11:19:41.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 monclient(hunting): picked mon.noname-b con 0x7f997406b730 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 monclient(hunting): picked mon.noname-a con 0x7f99740af360 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 monclient(hunting): picked mon.noname-c con 0x7f9974146320 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9974122180 con 0x7f99740af360 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9974123bf0 con 0x7f997406b730 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9974123e60 con 0x7f9974146320 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 monclient(hunting): authenticate will time out at 844.255285s 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f9972ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f997406b730 0x7f9974121a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44770/0 (socket says 172.21.15.172:44770) 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f9972ffd640 1 -- 172.21.15.172:0/3970391773 learned_addr learned my addr 172.21.15.172:0/3970391773 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 client.?.objecter ms_handle_connect 0x7f997406b730 2024-03-29T11:19:41.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 client.?.objecter ms_handle_connect 0x7f99740af360 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 client.?.objecter ms_handle_connect 0x7f9974146320 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723953243 0 0) 0x7f9974123bf0 con 0x7f997406b730 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9960004eb0 con 0x7f997406b730 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176614845 0 0) 0x7f9974122180 con 0x7f99740af360 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9974123bf0 con 0x7f99740af360 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860374313 0 0) 0x7f9974123e60 con 0x7f9974146320 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9974122180 con 0x7f9974146320 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2135997567 0 0) 0x7f9960004eb0 con 0x7f997406b730 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9974123e60 con 0x7f997406b730 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9964001a10 con 0x7f997406b730 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:41.988 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3745139746 0 0) 0x7f9974123bf0 con 0x7f99740af360 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9960004eb0 con 0x7f99740af360 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9968001b80 con 0x7f99740af360 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2336402877 0 0) 0x7f9974122180 con 0x7f9974146320 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9974123bf0 con 0x7f9974146320 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f996c001a50 con 0x7f9974146320 2024-03-29T11:19:41.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 385680121 0 0) 0x7f9974123e60 con 0x7f997406b730 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 >> v1:172.21.15.59:6790/0 conn(0x7f9974146320 legacy=0x7f99741d76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 >> v1:172.21.15.59:6789/0 conn(0x7f99740af360 legacy=0x7f99741d6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 monclient: found mon.b 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99741229e0 con 0x7f997406b730 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f995b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 monclient: _renew_subs 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 1 -- 172.21.15.172:0/3970391773 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9974141610 con 0x7f997406b730 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 monclient: _renew_subs 2024-03-29T11:19:41.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 1 -- 172.21.15.172:0/3970391773 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99741e2cc0 con 0x7f997406b730 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.996+0000 7f997926e640 1 librados: init done 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9964001c70 con 0x7f997406b730 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9964003a60 con 0x7f997406b730 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f99717fa640 4 set_mon_vals no callback set 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 10 monclient: dump: 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:41.993 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9964004010 con 0x7f997406b730 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4101236966 0 0) 0x7f9964006980 con 0x7f997406b730 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 10 client.4147.objecter ms_dispatch 0x7f9974000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.997+0000 7f995b7fe640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.998+0000 7f997926e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.998+0000 7f997926e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.998+0000 7f997926e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.998+0000 7f997926e640 1 -- 172.21.15.172:0/3970391773 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9954005080 con 0x7f997406b730 2024-03-29T11:19:41.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:41.999+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9964006390 con 0x7f997406b730 2024-03-29T11:19:41.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.000+0000 7f995b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:41.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.000+0000 7f995b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:42.050 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.072+0000 7f997926e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3", "id": 6}] 2024-03-29T11:19:42.050 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.072+0000 7f997926e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3", "id": 6}] 2024-03-29T11:19:42.050 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.072+0000 7f997926e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:42.050 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.072+0000 7f997926e640 1 -- 172.21.15.172:0/3970391773 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3", "id": 6} v 0) v1 -- 0x7f99540058b0 con 0x7f997406b730 2024-03-29T11:19:42.054 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.076+0000 7f995b7fe640 1 -- 172.21.15.172:0/3970391773 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1201495484 0 1113811849) 0x7f99640065b0 con 0x7f997406b730 2024-03-29T11:19:42.054 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.076+0000 7f995b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3", "id": 6}] 2024-03-29T11:19:42.054 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.076+0000 7f995b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:42.054 INFO:teuthology.orchestra.run.smithi172.stdout:6 2024-03-29T11:19:42.055 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.077+0000 7f997926e640 10 librados: watch_flush enter 2024-03-29T11:19:42.055 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.077+0000 7f997926e640 10 librados: watch_flush exit 2024-03-29T11:19:42.055 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.077+0000 7f997926e640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:19:42.055 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.077+0000 7f997926e640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:19:42.055 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.077+0000 7f997926e640 10 monclient: shutdown 2024-03-29T11:19:42.055 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.077+0000 7f997926e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:42.055 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.077+0000 7f997926e640 1 -- 172.21.15.172:0/3970391773 >> v1:172.21.15.172:6789/0 conn(0x7f997406b730 legacy=0x7f9974121a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.055 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.077+0000 7f997926e640 1 -- 172.21.15.172:0/3970391773 shutdown_connections 2024-03-29T11:19:42.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.077+0000 7f997926e640 1 -- 172.21.15.172:0/3970391773 >> 172.21.15.172:0/3970391773 conn(0x7f99740b58f0 msgr2=0x7f997409a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:42.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.078+0000 7f997926e640 1 -- 172.21.15.172:0/3970391773 shutdown_connections 2024-03-29T11:19:42.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.078+0000 7f997926e640 1 -- 172.21.15.172:0/3970391773 wait complete. 2024-03-29T11:19:42.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.078+0000 7f997926e640 1 librados: shutdown 2024-03-29T11:19:42.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new ee31832e-2287-4b25-b133-3af8315e4497 7 2024-03-29T11:19:42.070 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:42.093+0000 7f1c8c80f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:42.073 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:42.097+0000 7fa3c7432040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:42.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.188+0000 7fe8af59e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:42.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.188+0000 7fe8af59e640 10 monclient: build_initial_monmap 2024-03-29T11:19:42.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.188+0000 7fe8af59e640 10 monclient: monmap: 2024-03-29T11:19:42.165 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:42.166 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.166 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:42.190032+0000 2024-03-29T11:19:42.166 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:42.190032+0000 2024-03-29T11:19:42.166 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:42.166 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:42.166 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:42.166 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:42.166 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:42.166 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:42.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 1 Processor -- start 2024-03-29T11:19:42.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 1 -- start start 2024-03-29T11:19:42.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 10 monclient: init 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 10 monclient(hunting): picked mon.noname-a con 0x7fe8b0145610 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 10 monclient(hunting): picked mon.noname-c con 0x7fe8b0146050 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 10 monclient(hunting): picked mon.noname-b con 0x7fe8b013c030 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b00aacc0 con 0x7fe8b0145610 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b00b2720 con 0x7fe8b0146050 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b00aaf30 con 0x7fe8b013c030 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.189+0000 7fe8af59e640 10 monclient(hunting): authenticate will time out at 844.449103s 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8add9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe8b0146050 0x7fe8b013b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44786/0 (socket says 172.21.15.172:44786) 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8add9b640 1 -- 172.21.15.172:0/1632588400 learned_addr learned my addr 172.21.15.172:0/1632588400 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209858291 0 0) 0x7fe8b00b2720 con 0x7fe8b0146050 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe898003520 con 0x7fe8b0146050 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747792195 0 0) 0x7fe8b00aaf30 con 0x7fe8b013c030 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b00b2720 con 0x7fe8b013c030 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162997173 0 0) 0x7fe8b00aacc0 con 0x7fe8b0145610 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b00aaf30 con 0x7fe8b0145610 2024-03-29T11:19:42.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3989286116 0 0) 0x7fe898003520 con 0x7fe8b0146050 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 >> v1:172.21.15.59:6790/0 conn(0x7fe8b013c030 legacy=0x7fe8b014d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 >> v1:172.21.15.59:6789/0 conn(0x7fe8b0145610 legacy=0x7fe8b00b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 monclient: found mon.noname-c 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b00a9b20 con 0x7fe8b0146050 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient: _finish_auth 0 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe8a4001690 con 0x7fe8b0146050 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient: dump: 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:42.177 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8af59e640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8af59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8a4001870 con 0x7fe8b0146050 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 1 -- 172.21.15.172:0/1632588400 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe8a4001c00 con 0x7fe8b0146050 2024-03-29T11:19:42.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8ad59a640 10 monclient: dump: 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8af59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.190+0000 7fe8af59e640 4 set_mon_vals no callback set 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.191+0000 7fe8af59e640 10 monclient: shutdown 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.191+0000 7fe8af59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.191+0000 7fe8af59e640 1 -- 172.21.15.172:0/1632588400 >> v1:172.21.15.172:6789/0 conn(0x7fe8b0146050 legacy=0x7fe8b013b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.191+0000 7fe8af59e640 1 -- 172.21.15.172:0/1632588400 shutdown_connections 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.191+0000 7fe8af59e640 1 -- 172.21.15.172:0/1632588400 >> 172.21.15.172:0/1632588400 conn(0x7fe8b00726e0 msgr2=0x7fe8b0066920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.191+0000 7fe8af59e640 1 -- 172.21.15.172:0/1632588400 shutdown_connections 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.191+0000 7fe8af59e640 1 -- 172.21.15.172:0/1632588400 wait complete. 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.192+0000 7fe8af59e640 10 monclient: build_initial_monmap 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.192+0000 7fe8af59e640 10 monclient: monmap: 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 0 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 0.000000 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:created 0.000000 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:42.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.192+0000 7fe8af59e640 1 librados: starting msgr at 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.192+0000 7fe8af59e640 1 librados: starting objecter 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.192+0000 7fe8af59e640 1 Processor -- start 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.192+0000 7fe8af59e640 1 -- start start 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.192+0000 7fe8af59e640 1 librados: setting wanted keys 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.192+0000 7fe8af59e640 1 librados: calling monclient init 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.192+0000 7fe8af59e640 10 monclient: init 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8af59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8af59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8af59e640 10 monclient(hunting): picked mon.noname-c con 0x7fe8b013c030 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8af59e640 10 monclient(hunting): picked mon.noname-a con 0x7fe8b0145610 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8af59e640 10 monclient(hunting): picked mon.noname-b con 0x7fe8b0146050 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8af59e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b00acd00 con 0x7fe8b0145610 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8af59e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b00acfe0 con 0x7fe8b0146050 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8af59e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b013f340 con 0x7fe8b013c030 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8af59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8af59e640 10 monclient(hunting): authenticate will time out at 844.452380s 2024-03-29T11:19:42.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8aed9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe8b0146050 0x7fe8b01d7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44802/0 (socket says 172.21.15.172:44802) 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8aed9d640 1 -- 172.21.15.172:0/3183994854 learned_addr learned my addr 172.21.15.172:0/3183994854 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe8ae59c640 1 --1- 172.21.15.172:0/3183994854 >> v1:172.21.15.59:6790/0 conn(0x7fe8b013c030 0x7fe8b0137dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.172:40006/0 (socket says 172.21.15.172:40006) 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 client.?.objecter ms_handle_connect 0x7fe8b0146050 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 client.?.objecter ms_handle_connect 0x7fe8b0145610 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 client.?.objecter ms_handle_connect 0x7fe8b013c030 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309641424 0 0) 0x7fe8b00acfe0 con 0x7fe8b0146050 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe890004eb0 con 0x7fe8b0146050 2024-03-29T11:19:42.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591843820 0 0) 0x7fe8b00acd00 con 0x7fe8b0145610 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b00acfe0 con 0x7fe8b0145610 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600010584 0 0) 0x7fe8b013f340 con 0x7fe8b013c030 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b00acd00 con 0x7fe8b013c030 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3006723272 0 0) 0x7fe890004eb0 con 0x7fe8b0146050 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8b013f340 con 0x7fe8b0146050 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe8a8001cb0 con 0x7fe8b0146050 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): dump: 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 481280165 0 0) 0x7fe8b00acd00 con 0x7fe8b013c030 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe890004eb0 con 0x7fe8b013c030 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe8a0001870 con 0x7fe8b013c030 2024-03-29T11:19:42.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): dump: 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1810680184 0 0) 0x7fe8b00acfe0 con 0x7fe8b0145610 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8b00acd00 con 0x7fe8b0145610 2024-03-29T11:19:42.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe8a4001b80 con 0x7fe8b0145610 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 10 monclient(hunting): dump: 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:42.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3598740189 0 0) 0x7fe8b013f340 con 0x7fe8b0146050 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.193+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 >> v1:172.21.15.59:6790/0 conn(0x7fe8b013c030 legacy=0x7fe8b0137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 >> v1:172.21.15.59:6789/0 conn(0x7fe8b0145610 legacy=0x7fe8b01d6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 1 monclient: found mon.b 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b013f010 con 0x7fe8b0146050 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 10 monclient: _finish_auth 0 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 10 monclient: _renew_subs 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 1 -- 172.21.15.172:0/3183994854 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b00abc30 con 0x7fe8b0146050 2024-03-29T11:19:42.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 10 monclient: _renew_subs 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 1 -- 172.21.15.172:0/3183994854 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b00ab7a0 con 0x7fe8b0146050 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 1 librados: init done 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8a8001e90 con 0x7fe8b0146050 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8acd99640 4 set_mon_vals no callback set 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe8a8003c40 con 0x7fe8b0146050 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 10 monclient: dump: 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:epoch 1 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.188 INFO:teuthology.orchestra.run.smithi172.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:election_strategy: 1 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe8a80041f0 con 0x7fe8b0146050 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2295041298 0 0) 0x7fe8a8006c30 con 0x7fe8b0146050 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 10 client.4153.objecter ms_dispatch 0x7fe8b0000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe89effd640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:42.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.194+0000 7fe8af59e640 1 -- 172.21.15.172:0/3183994854 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b01d6ce0 con 0x7fe8b0146050 2024-03-29T11:19:42.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.198+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe8a80046d0 con 0x7fe8b0146050 2024-03-29T11:19:42.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.198+0000 7fe89effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.198+0000 7fe89effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:42.214 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:42.237+0000 7f1c8c80f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:42.215 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:42.238+0000 7fa3c7432040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:42.248 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.271+0000 7fe8af59e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ee31832e-2287-4b25-b133-3af8315e4497", "id": 7}] 2024-03-29T11:19:42.248 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.271+0000 7fe8af59e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ee31832e-2287-4b25-b133-3af8315e4497", "id": 7}] 2024-03-29T11:19:42.248 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.271+0000 7fe8af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:42.248 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.271+0000 7fe8af59e640 1 -- 172.21.15.172:0/3183994854 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ee31832e-2287-4b25-b133-3af8315e4497", "id": 7} v 0) v1 -- 0x7fe8b00758c0 con 0x7fe8b0146050 2024-03-29T11:19:42.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.276+0000 7fe89effd640 1 -- 172.21.15.172:0/3183994854 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ee31832e-2287-4b25-b133-3af8315e4497", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2526049292 0 1371665406) 0x7fe8a8006700 con 0x7fe8b0146050 2024-03-29T11:19:42.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.276+0000 7fe89effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ee31832e-2287-4b25-b133-3af8315e4497", "id": 7}] 2024-03-29T11:19:42.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.276+0000 7fe89effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:42.253 INFO:teuthology.orchestra.run.smithi172.stdout:7 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 10 librados: watch_flush enter 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 10 librados: watch_flush exit 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 10 client.4153.objecter successfully canceled tick 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 10 monclient: shutdown 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 1 -- 172.21.15.172:0/3183994854 >> v1:172.21.15.172:6789/0 conn(0x7fe8b0146050 legacy=0x7fe8b01d7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 1 -- 172.21.15.172:0/3183994854 shutdown_connections 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 1 -- 172.21.15.172:0/3183994854 >> 172.21.15.172:0/3183994854 conn(0x7fe8b00726e0 msgr2=0x7fe8b0144b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 1 -- 172.21.15.172:0/3183994854 shutdown_connections 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 1 -- 172.21.15.172:0/3183994854 wait complete. 2024-03-29T11:19:42.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-29T11:19:42.277+0000 7fe8af59e640 1 librados: shutdown 2024-03-29T11:19:42.264 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:19:42.264 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:19:42.267 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:19:42.267 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:19:42.267 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:19:42.269 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:19:42.269 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:19:42.269 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:19:42.271 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:19:42.272 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:19:42.272 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:19:42.274 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:19:42.274 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:19:42.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:19:42.306 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:19:42.306 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:19:42.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:19:42.308 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:19:42.309 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:19:42.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:19:42.311 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:19:42.311 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:19:42.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:19:42.313 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:19:42.313 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:42.337 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:42.361+0000 7fa3c7432040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:42.362 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:42.385+0000 7f1c8c80f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:42.399 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:42.423+0000 7fa3c7432040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:42.416 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:42.438+0000 7f1c8c80f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:42.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.454+0000 7fc0c44b3640 10 monclient: get_monmap_and_config 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.454+0000 7fc0c44b3640 10 monclient: build_initial_monmap 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.454+0000 7fc0c44b3640 10 monclient: monmap: 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:42.455329+0000 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:42.455329+0000 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:42.431 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:42.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.455+0000 7fc0c44b3640 1 Processor -- start 2024-03-29T11:19:42.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.455+0000 7fc0c44b3640 1 -- start start 2024-03-29T11:19:42.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.455+0000 7fc0c44b3640 10 monclient: init 2024-03-29T11:19:42.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.455+0000 7fc0c44b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.455+0000 7fc0c44b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.455+0000 7fc0c44b3640 10 monclient(hunting): picked mon.noname-c con 0x7fc0bc0b8240 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.455+0000 7fc0c44b3640 10 monclient(hunting): picked mon.noname-b con 0x7fc0bc0b1f60 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c44b3640 10 monclient(hunting): picked mon.noname-a con 0x7fc0bc09c8c0 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c1a4e640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fc0bc0b1f60 0x7fc0bc09c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:47654/0 (socket says 172.21.15.59:47654) 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c1a4e640 1 -- 172.21.15.59:0/3243877175 learned_addr learned my addr 172.21.15.59:0/3243877175 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3243877175 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc09bd80 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3243877175 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc09b960 con 0x7fc0bc0b8240 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3243877175 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc0b2ec0 con 0x7fc0bc0b1f60 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c44b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c44b3640 10 monclient(hunting): authenticate will time out at 832.569849s 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036891481 0 0) 0x7fc0bc09bd80 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a8003520 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530586854 0 0) 0x7fc0bc0b2ec0 con 0x7fc0bc0b1f60 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): my global_id is 4175 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0bc09bd80 con 0x7fc0bc0b1f60 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748636691 0 0) 0x7fc0bc09b960 con 0x7fc0bc0b8240 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0bc0b2ec0 con 0x7fc0bc0b8240 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3165854749 0 0) 0x7fc0a8003520 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 >> v1:172.21.15.59:6790/0 conn(0x7fc0bc0b1f60 legacy=0x7fc0bc09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 >> v1:172.21.15.172:6789/0 conn(0x7fc0bc0b8240 legacy=0x7fc0bc13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient: _finish_hunting 0 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 monclient: found mon.noname-a 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc13fb60 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient: _finish_auth 0 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient: _check_auth_tickets 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc0b4001a10 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient: dump: 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c44b3640 5 monclient: authenticate success, global_id 4185 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c44b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:42.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0b4001bf0 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 1 -- 172.21.15.59:0/3243877175 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc0b4002000 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c124d640 10 monclient: dump: 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.440 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c44b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.456+0000 7fc0c44b3640 4 set_mon_vals no callback set 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.457+0000 7fc0c44b3640 10 monclient: shutdown 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.457+0000 7fc0c44b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.457+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3243877175 >> v1:172.21.15.59:6789/0 conn(0x7fc0bc09c8c0 legacy=0x7fc0bc14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.457+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3243877175 shutdown_connections 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.457+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3243877175 >> 172.21.15.59:0/3243877175 conn(0x7fc0bc066b00 msgr2=0x7fc0bc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.457+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3243877175 shutdown_connections 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.457+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3243877175 wait complete. 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.458+0000 7fc0c44b3640 10 monclient: build_initial_monmap 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.458+0000 7fc0c44b3640 10 monclient: monmap: 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2024-03-29T11:19:42.441 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.459+0000 7fc0c44b3640 1 librados: starting msgr at 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.459+0000 7fc0c44b3640 1 librados: starting objecter 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.459+0000 7fc0c44b3640 1 Processor -- start 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.459+0000 7fc0c44b3640 1 -- start start 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.459+0000 7fc0c44b3640 1 librados: setting wanted keys 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.459+0000 7fc0c44b3640 1 librados: calling monclient init 2024-03-29T11:19:42.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.459+0000 7fc0c44b3640 10 monclient: init 2024-03-29T11:19:42.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c44b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:42.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c44b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:42.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c44b3640 10 monclient(hunting): picked mon.noname-a con 0x7fc0bc09c8c0 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:42.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c44b3640 10 monclient(hunting): picked mon.noname-c con 0x7fc0bc0b8240 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:42.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c44b3640 10 monclient(hunting): picked mon.noname-b con 0x7fc0bc14e530 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c44b3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc06ece0 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c44b3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc06efc0 con 0x7fc0bc14e530 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c44b3640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc06daf0 con 0x7fc0bc0b8240 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c44b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c44b3640 10 monclient(hunting): authenticate will time out at 832.573889s 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c224f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc0bc09c8c0 0x7fc0bc14bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50852/0 (socket says 172.21.15.59:50852) 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0c224f640 1 -- 172.21.15.59:0/3231675140 learned_addr learned my addr 172.21.15.59:0/3231675140 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 client.?.objecter ms_handle_connect 0x7fc0bc09c8c0 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 client.?.objecter ms_handle_connect 0x7fc0bc0b8240 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297892860 0 0) 0x7fc0bc06ece0 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a0004eb0 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 client.?.objecter ms_handle_connect 0x7fc0bc14e530 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955922614 0 0) 0x7fc0bc06daf0 con 0x7fc0bc0b8240 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): my global_id is 4178 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0bc06ece0 con 0x7fc0bc0b8240 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252056287 0 0) 0x7fc0bc06efc0 con 0x7fc0bc14e530 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0bc06daf0 con 0x7fc0bc14e530 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2700356627 0 0) 0x7fc0a0004eb0 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0bc06efc0 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc0b8001a10 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.449 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1226106136 0 0) 0x7fc0bc06ece0 con 0x7fc0bc0b8240 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0a0004eb0 con 0x7fc0bc0b8240 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc0ac0018b0 con 0x7fc0bc0b8240 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.450 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3587943731 0 0) 0x7fc0bc06daf0 con 0x7fc0bc14e530 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0bc06ece0 con 0x7fc0bc14e530 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc0b4001f00 con 0x7fc0bc14e530 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.460+0000 7fc0b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3160597942 0 0) 0x7fc0bc06efc0 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 >> v1:172.21.15.59:6790/0 conn(0x7fc0bc0b8240 legacy=0x7fc0bc14c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 >> v1:172.21.15.172:6789/0 conn(0x7fc0bc14e530 legacy=0x7fc0bc1e61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 1 monclient: found mon.a 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc06dc80 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0c44b3640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0c44b3640 10 monclient: _renew_subs 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0c44b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3231675140 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0bc06cfe0 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0c44b3640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0c44b3640 10 monclient: _renew_subs 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0c44b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3231675140 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0bc1e78c0 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0c44b3640 1 librados: init done 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0b8001c70 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc0b8003a60 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b3fff640 4 set_mon_vals no callback set 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 10 monclient: dump: 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc0b8004010 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fc0b8006a60 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 10 client.4188.objecter ms_dispatch 0x7fc0bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:42.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:42.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0b2ffd640 20 client.4188.objecter dump_active .. 0 homeless 2024-03-29T11:19:42.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0c44b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.461+0000 7fc0c44b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.462+0000 7fc0c44b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:42.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.462+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3231675140 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a4005080 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.463+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc0b8006390 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.463+0000 7fc0b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.463+0000 7fc0b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:42.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.526+0000 7fc0c44b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.526+0000 7fc0c44b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.526+0000 7fc0c44b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:42.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.526+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3231675140 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0a4005330 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.527+0000 7fc0b2ffd640 1 -- 172.21.15.59:0/3231675140 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fc0b80065b0 con 0x7fc0bc09c8c0 2024-03-29T11:19:42.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.527+0000 7fc0b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.527+0000 7fc0b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:42.504 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-29T11:19:42.504 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.527+0000 7fc0c44b3640 10 librados: watch_flush enter 2024-03-29T11:19:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.527+0000 7fc0c44b3640 10 librados: watch_flush exit 2024-03-29T11:19:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.528+0000 7fc0c44b3640 20 client.4188.objecter shutdown clearing up homeless session... 2024-03-29T11:19:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.528+0000 7fc0c44b3640 10 client.4188.objecter successfully canceled tick 2024-03-29T11:19:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.528+0000 7fc0c44b3640 10 monclient: shutdown 2024-03-29T11:19:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.528+0000 7fc0c44b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.528+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3231675140 >> v1:172.21.15.59:6789/0 conn(0x7fc0bc09c8c0 legacy=0x7fc0bc14bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.528+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3231675140 shutdown_connections 2024-03-29T11:19:42.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.528+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3231675140 >> 172.21.15.59:0/3231675140 conn(0x7fc0bc066b00 msgr2=0x7fc0bc0b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:42.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.528+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3231675140 shutdown_connections 2024-03-29T11:19:42.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.528+0000 7fc0c44b3640 1 -- 172.21.15.59:0/3231675140 wait complete. 2024-03-29T11:19:42.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-29T11:19:42.528+0000 7fc0c44b3640 1 librados: shutdown 2024-03-29T11:19:42.514 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:19:42.515 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:19:42.521 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:42.544+0000 7fa3c7432040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:42.529 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:42.552+0000 7f1c8c80f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:42.619 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:42.643+0000 7fa3c7432040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:42.623 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:42.646+0000 7f1c8c80f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:42.780 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:42.802+0000 7f1c8c80f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:42.780 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:42.804+0000 7fa3c7432040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:42.821 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:42.844+0000 7f1c8c80f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:42.824 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:42.848+0000 7fa3c7432040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:42.876 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-03-29T11:19:42.899+0000 7fe437a0c640 -1 Falling back to public interface 2024-03-29T11:19:42.880 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-03-29T11:19:42.904+0000 7f9b4d8c2640 -1 Falling back to public interface 2024-03-29T11:19:42.888 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-03-29T11:19:42.912+0000 7f7422b24640 -1 Falling back to public interface 2024-03-29T11:19:42.897 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-03-29T11:19:42.921+0000 7f00dd80f640 -1 Falling back to public interface 2024-03-29T11:19:42.905 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-29T11:19:42.928+0000 7fbd0fd24640 -1 Falling back to public interface 2024-03-29T11:19:42.906 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-29T11:19:42.929+0000 7f8fb0523640 -1 Falling back to public interface 2024-03-29T11:19:42.909 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-29T11:19:42.932+0000 7f005718a640 -1 Falling back to public interface 2024-03-29T11:19:42.923 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-29T11:19:42.946+0000 7f0198d38640 -1 Falling back to public interface 2024-03-29T11:19:42.978 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:43.001+0000 7f1c8c80f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:42.988 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:43.012+0000 7fa3c7432040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:43.321 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:43.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.435+0000 7ff159585640 10 monclient: get_monmap_and_config 2024-03-29T11:19:43.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.435+0000 7ff159585640 10 monclient: build_initial_monmap 2024-03-29T11:19:43.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.435+0000 7ff159585640 10 monclient: monmap: 2024-03-29T11:19:43.412 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:19:43.412 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:43.412 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:43.436059+0000 2024-03-29T11:19:43.412 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:43.436059+0000 2024-03-29T11:19:43.412 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:43.412 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:43.412 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:43.412 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:43.412 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:43.413 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:43.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.435+0000 7ff159585640 1 Processor -- start 2024-03-29T11:19:43.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.435+0000 7ff159585640 1 -- start start 2024-03-29T11:19:43.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.435+0000 7ff159585640 10 monclient: init 2024-03-29T11:19:43.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff159585640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:43.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff159585640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:43.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff159585640 10 monclient(hunting): picked mon.noname-c con 0x7ff1541422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:43.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff159585640 10 monclient(hunting): picked mon.noname-b con 0x7ff154142690 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:43.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff159585640 10 monclient(hunting): picked mon.noname-a con 0x7ff1540b17d0 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:43.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff159585640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff154142c80 con 0x7ff1540b17d0 2024-03-29T11:19:43.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff159585640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1540aae40 con 0x7ff1541422b0 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff159585640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15413fbd0 con 0x7ff154142690 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff159585640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff159585640 10 monclient(hunting): authenticate will time out at 833.550253s 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff1527fc640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7ff154142690 0x7ff1540b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:59584/0 (socket says 172.21.15.59:59584) 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff1527fc640 1 -- 172.21.15.59:0/896307181 learned_addr learned my addr 172.21.15.59:0/896307181 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff1537fe640 1 --1- 172.21.15.59:0/896307181 >> v1:172.21.15.59:6789/0 conn(0x7ff1540b17d0 0x7ff1540b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55680/0 (socket says 172.21.15.59:55680) 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 1 -- 172.21.15.59:0/896307181 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960375458 0 0) 0x7ff15413fbd0 con 0x7ff154142690 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 1 -- 172.21.15.59:0/896307181 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13c003520 con 0x7ff154142690 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 1 -- 172.21.15.59:0/896307181 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194604924 0 0) 0x7ff154142c80 con 0x7ff1540b17d0 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.436+0000 7ff151ffb640 1 -- 172.21.15.59:0/896307181 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff15413fbd0 con 0x7ff1540b17d0 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 1 -- 172.21.15.59:0/896307181 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3178368331 0 0) 0x7ff15413fbd0 con 0x7ff1540b17d0 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 1 -- 172.21.15.59:0/896307181 >> v1:172.21.15.59:6790/0 conn(0x7ff154142690 legacy=0x7ff1540b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 1 -- 172.21.15.59:0/896307181 >> v1:172.21.15.172:6789/0 conn(0x7ff1541422b0 legacy=0x7ff15413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 1 monclient: found mon.noname-a 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 1 -- 172.21.15.59:0/896307181 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff154151940 con 0x7ff1540b17d0 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:43.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 1 -- 172.21.15.59:0/896307181 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff14c001920 con 0x7ff1540b17d0 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 10 monclient: dump: 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:43.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 1 -- 172.21.15.59:0/896307181 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff14c001a80 con 0x7ff1540b17d0 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 1 -- 172.21.15.59:0/896307181 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff14c001e50 con 0x7ff1540b17d0 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 4 set_mon_vals no callback set 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff151ffb640 10 monclient: dump: 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:43.424 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 10 monclient: shutdown 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 1 -- 172.21.15.59:0/896307181 >> v1:172.21.15.59:6789/0 conn(0x7ff1540b17d0 legacy=0x7ff1540b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 1 -- 172.21.15.59:0/896307181 shutdown_connections 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 1 -- 172.21.15.59:0/896307181 >> 172.21.15.59:0/896307181 conn(0x7ff154066a00 msgr2=0x7ff154066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 1 -- 172.21.15.59:0/896307181 shutdown_connections 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.437+0000 7ff159585640 1 -- 172.21.15.59:0/896307181 wait complete. 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.438+0000 7ff159585640 10 monclient: build_initial_monmap 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.438+0000 7ff159585640 10 monclient: monmap: 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:43.425 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.438+0000 7ff159585640 1 librados: starting msgr at 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.438+0000 7ff159585640 1 librados: starting objecter 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 1 Processor -- start 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 1 -- start start 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 1 librados: setting wanted keys 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 1 librados: calling monclient init 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 10 monclient: init 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 10 monclient(hunting): picked mon.noname-a con 0x7ff1540b17d0 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 10 monclient(hunting): picked mon.noname-b con 0x7ff1541422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:43.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 10 monclient(hunting): picked mon.noname-c con 0x7ff154142690 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15412da80 con 0x7ff1540b17d0 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1540b7fb0 con 0x7ff1541422b0 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1540ab630 con 0x7ff154142690 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff159585640 10 monclient(hunting): authenticate will time out at 833.553275s 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff152ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff1540b17d0 0x7ff1540b74e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55694/0 (socket says 172.21.15.59:55694) 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff1537fe640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7ff154142690 0x7ff1541e1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:59592/0 (socket says 172.21.15.59:59592) 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff152ffd640 1 -- 172.21.15.59:0/3611655885 learned_addr learned my addr 172.21.15.59:0/3611655885 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 client.?.objecter ms_handle_connect 0x7ff154142690 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 client.?.objecter ms_handle_connect 0x7ff1540b17d0 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873627137 0 0) 0x7ff1540ab630 con 0x7ff154142690 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff140004eb0 con 0x7ff154142690 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 client.?.objecter ms_handle_connect 0x7ff1541422b0 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630333861 0 0) 0x7ff15412da80 con 0x7ff1540b17d0 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.439+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1540ab630 con 0x7ff1540b17d0 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183018356 0 0) 0x7ff1540b7fb0 con 0x7ff1541422b0 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:19:43.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff15412da80 con 0x7ff1541422b0 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 269755533 0 0) 0x7ff140004eb0 con 0x7ff154142690 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1540b7fb0 con 0x7ff154142690 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff14c0017e0 con 0x7ff154142690 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:43.429 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1451753046 0 0) 0x7ff1540ab630 con 0x7ff1540b17d0 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff140004eb0 con 0x7ff1540b17d0 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff1440019b0 con 0x7ff1540b17d0 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:43.430 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2576237745 0 0) 0x7ff15412da80 con 0x7ff1541422b0 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1540ab630 con 0x7ff1541422b0 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff148001750 con 0x7ff1541422b0 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:43.431 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 952333599 0 0) 0x7ff1540b7fb0 con 0x7ff154142690 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 >> v1:172.21.15.172:6789/0 conn(0x7ff1541422b0 legacy=0x7ff1540b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 >> v1:172.21.15.59:6789/0 conn(0x7ff1540b17d0 legacy=0x7ff1540b74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 monclient: found mon.c 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540ab910 con 0x7ff154142690 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:43.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff159585640 5 monclient: authenticate success, global_id 4184 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff159585640 10 monclient: _renew_subs 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff159585640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff159585640 1 -- 172.21.15.59:0/3611655885 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1540abc50 con 0x7ff154142690 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff14c001250 con 0x7ff154142690 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff159585640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff159585640 10 monclient: _renew_subs 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff159585640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff159585640 1 -- 172.21.15.59:0/3611655885 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1541313f0 con 0x7ff154142690 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff14c004fb0 con 0x7ff154142690 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff159585640 1 librados: init done 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff1517fa640 4 set_mon_vals no callback set 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 monclient: dump: 2024-03-29T11:19:43.433 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff14c0054a0 con 0x7ff154142690 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7ff14c006bd0 con 0x7ff154142690 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 10 client.4184.objecter ms_dispatch 0x7ff154000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:43.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:43.435 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.440+0000 7ff13b7fe640 20 client.4184.objecter dump_active .. 0 homeless 2024-03-29T11:19:43.435 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.441+0000 7ff159585640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:43.435 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.441+0000 7ff159585640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:43.435 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.441+0000 7ff159585640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:19:43.435 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.441+0000 7ff159585640 1 -- 172.21.15.59:0/3611655885 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1540b1340 con 0x7ff154142690 2024-03-29T11:19:43.435 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.443+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff14c006610 con 0x7ff154142690 2024-03-29T11:19:43.435 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.443+0000 7ff13b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:43.435 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.443+0000 7ff13b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:43.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.505+0000 7ff159585640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:43.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.505+0000 7ff159585640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:43.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.505+0000 7ff159585640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:19:43.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.505+0000 7ff159585640 1 -- 172.21.15.59:0/3611655885 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff154131740 con 0x7ff154142690 2024-03-29T11:19:43.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.506+0000 7ff13b7fe640 1 -- 172.21.15.59:0/3611655885 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7ff14c006860 con 0x7ff154142690 2024-03-29T11:19:43.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.506+0000 7ff13b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:43.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.506+0000 7ff13b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:43.483 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:19:43.483 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 10 librados: watch_flush enter 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 10 librados: watch_flush exit 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 20 client.4184.objecter shutdown clearing up homeless session... 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 10 client.4184.objecter successfully canceled tick 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 10 monclient: shutdown 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 1 -- 172.21.15.59:0/3611655885 >> v1:172.21.15.59:6790/0 conn(0x7ff154142690 legacy=0x7ff1541e1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 1 -- 172.21.15.59:0/3611655885 shutdown_connections 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 1 -- 172.21.15.59:0/3611655885 >> 172.21.15.59:0/3611655885 conn(0x7ff154066a00 msgr2=0x7ff15413dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 1 -- 172.21.15.59:0/3611655885 shutdown_connections 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 1 -- 172.21.15.59:0/3611655885 wait complete. 2024-03-29T11:19:43.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:43.507+0000 7ff159585640 1 librados: shutdown 2024-03-29T11:19:43.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:43.778 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:43.800+0000 7f1c8c80f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:43.829 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:43.853+0000 7fa3c7432040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:44.054 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:44.077+0000 7f1c8c80f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:44.105 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:44.128+0000 7f1c8c80f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:44.115 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:44.139+0000 7fa3c7432040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:44.168 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:44.192+0000 7fa3c7432040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:44.220 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:44.243+0000 7f1c8c80f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:44.293 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-29T11:19:44.317+0000 7fa3c7432040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:45.236 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:45.258+0000 7f1c2d026640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:19:45.236 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-29T11:19:45.259+0000 7f1c2d026640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:19:46.347 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-03-29T11:19:46.371+0000 7fe437a0c640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:19:46.357 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-29T11:19:46.380+0000 7fbd0fd24640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:19:46.396 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-03-29T11:19:46.420+0000 7f9b4d8c2640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:19:46.404 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-29T11:19:46.427+0000 7f8fb0523640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:19:46.411 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-03-29T11:19:46.435+0000 7f7422b24640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:19:46.412 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-29T11:19:46.434+0000 7f005718a640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:19:46.450 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-29T11:19:46.473+0000 7f0198d38640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:19:46.452 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-03-29T11:19:46.475+0000 7f00dd80f640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:19:50.300 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:50.452 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.475+0000 7f4effe8d640 10 monclient: get_monmap_and_config 2024-03-29T11:19:50.452 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.475+0000 7f4effe8d640 10 monclient: build_initial_monmap 2024-03-29T11:19:50.453 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.476+0000 7f4effe8d640 10 monclient: monmap: 2024-03-29T11:19:50.453 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:19:50.453 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:50.453 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:50.476762+0000 2024-03-29T11:19:50.453 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:50.476762+0000 2024-03-29T11:19:50.453 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:50.453 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:50.453 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:50.453 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:50.453 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:50.453 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:50.454 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.477+0000 7f4effe8d640 1 Processor -- start 2024-03-29T11:19:50.454 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.477+0000 7f4effe8d640 1 -- start start 2024-03-29T11:19:50.454 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.477+0000 7f4effe8d640 10 monclient: init 2024-03-29T11:19:50.454 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4effe8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:50.454 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4effe8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:50.454 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4effe8d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef81457f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:50.455 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4effe8d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef813f650 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:50.455 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4effe8d640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef80a9d70 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:50.455 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4effe8d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef80aaef0 con 0x7f4ef813f650 2024-03-29T11:19:50.455 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4effe8d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef8145bd0 con 0x7f4ef80a9d70 2024-03-29T11:19:50.455 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4effe8d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef8145eb0 con 0x7f4ef81457f0 2024-03-29T11:19:50.455 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4effe8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:50.455 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4effe8d640 10 monclient(hunting): authenticate will time out at 840.592449s 2024-03-29T11:19:50.455 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4efe42a640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4ef80a9d70 0x7f4ef814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:42690/0 (socket says 172.21.15.59:42690) 2024-03-29T11:19:50.455 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.478+0000 7f4efe42a640 1 -- 172.21.15.59:0/3579561336 learned_addr learned my addr 172.21.15.59:0/3579561336 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:19:50.456 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.479+0000 7f4efcc27640 1 -- 172.21.15.59:0/3579561336 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995407529 0 0) 0x7f4ef8145bd0 con 0x7f4ef80a9d70 2024-03-29T11:19:50.456 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.479+0000 7f4efcc27640 10 monclient(hunting): _negotiate 2024-03-29T11:19:50.456 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.479+0000 7f4efcc27640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:50.456 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.479+0000 7f4efcc27640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:50.456 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.479+0000 7f4efcc27640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:19:50.456 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.479+0000 7f4efcc27640 1 -- 172.21.15.59:0/3579561336 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee4003520 con 0x7f4ef80a9d70 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 1 -- 172.21.15.59:0/3579561336 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744345038 0 0) 0x7f4ef80aaef0 con 0x7f4ef813f650 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient(hunting): _negotiate 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient(hunting): my global_id is 4236 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 1 -- 172.21.15.59:0/3579561336 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef8145bd0 con 0x7f4ef813f650 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 1 -- 172.21.15.59:0/3579561336 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 88101786 0 0) 0x7f4ee4003520 con 0x7f4ef80a9d70 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 1 -- 172.21.15.59:0/3579561336 >> v1:172.21.15.59:6790/0 conn(0x7f4ef81457f0 legacy=0x7f4ef813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 1 -- 172.21.15.59:0/3579561336 >> v1:172.21.15.59:6789/0 conn(0x7f4ef813f650 legacy=0x7f4ef80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient: _finish_hunting 0 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 1 monclient: found mon.noname-c 2024-03-29T11:19:50.457 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 1 -- 172.21.15.59:0/3579561336 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef8146340 con 0x7f4ef80a9d70 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient: _finish_auth 0 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient: _check_auth_tickets 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4effe8d640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4effe8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 1 -- 172.21.15.59:0/3579561336 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4ef0001690 con 0x7f4ef80a9d70 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient: dump: 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:50.458 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4effe8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 1 -- 172.21.15.59:0/3579561336 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ef00018b0 con 0x7f4ef80a9d70 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4efcc27640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4effe8d640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:50.459 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.480+0000 7f4effe8d640 4 set_mon_vals no callback set 2024-03-29T11:19:50.466 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.481+0000 7f4efcc27640 1 -- 172.21.15.59:0/3579561336 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4ef0001c40 con 0x7f4ef80a9d70 2024-03-29T11:19:50.466 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.481+0000 7f4efcc27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.481+0000 7f4efcc27640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.481+0000 7f4efcc27640 10 monclient: dump: 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.481+0000 7f4effe8d640 10 monclient: shutdown 2024-03-29T11:19:50.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.481+0000 7f4effe8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.481+0000 7f4effe8d640 1 -- 172.21.15.59:0/3579561336 >> v1:172.21.15.172:6789/0 conn(0x7f4ef80a9d70 legacy=0x7f4ef814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.481+0000 7f4effe8d640 1 -- 172.21.15.59:0/3579561336 shutdown_connections 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.481+0000 7f4effe8d640 1 -- 172.21.15.59:0/3579561336 >> 172.21.15.59:0/3579561336 conn(0x7f4ef8066a00 msgr2=0x7f4ef8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.481+0000 7f4effe8d640 1 -- 172.21.15.59:0/3579561336 shutdown_connections 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.482+0000 7f4effe8d640 1 -- 172.21.15.59:0/3579561336 wait complete. 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.483+0000 7f4effe8d640 10 monclient: build_initial_monmap 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.483+0000 7f4effe8d640 10 monclient: monmap: 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:50.468 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.483+0000 7f4effe8d640 1 librados: starting msgr at 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.483+0000 7f4effe8d640 1 librados: starting objecter 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.483+0000 7f4effe8d640 1 Processor -- start 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.484+0000 7f4effe8d640 1 -- start start 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.484+0000 7f4effe8d640 1 librados: setting wanted keys 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.484+0000 7f4effe8d640 1 librados: calling monclient init 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.484+0000 7f4effe8d640 10 monclient: init 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4effe8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4effe8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4effe8d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef80a9d70 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4effe8d640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef813f650 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4effe8d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef81457f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:50.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4effe8d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef81cbd90 con 0x7f4ef81457f0 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4effe8d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef81ccef0 con 0x7f4ef80a9d70 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4effe8d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef8142a60 con 0x7f4ef813f650 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4effe8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4effe8d640 10 monclient(hunting): authenticate will time out at 840.599072s 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4efdc29640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4ef80a9d70 0x7f4ef81cb340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:42850/0 (socket says 172.21.15.59:42850) 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4efdc29640 1 -- 172.21.15.59:0/726596005 learned_addr learned my addr 172.21.15.59:0/726596005 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4eee7fc640 10 client.?.objecter ms_handle_connect 0x7f4ef80a9d70 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.485+0000 7f4eee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 client.?.objecter ms_handle_connect 0x7f4ef813f650 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891953019 0 0) 0x7f4ef81ccef0 con 0x7f4ef80a9d70 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:50.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4edc004eb0 con 0x7f4ef80a9d70 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 client.?.objecter ms_handle_connect 0x7f4ef81457f0 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218423176 0 0) 0x7f4ef8142a60 con 0x7f4ef813f650 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef81ccef0 con 0x7f4ef813f650 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 220560214 0 0) 0x7f4edc004eb0 con 0x7f4ef80a9d70 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ef8142a60 con 0x7f4ef80a9d70 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755151932 0 0) 0x7f4ef81cbd90 con 0x7f4ef81457f0 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:50.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4edc004eb0 con 0x7f4ef81457f0 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4ee8001cb0 con 0x7f4ef80a9d70 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:50.472 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2468865485 0 0) 0x7f4ef81ccef0 con 0x7f4ef813f650 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.486+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ef81cbd90 con 0x7f4ef813f650 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4ef4001930 con 0x7f4ef813f650 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:50.473 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1075043732 0 0) 0x7f4ef8142a60 con 0x7f4ef80a9d70 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 >> v1:172.21.15.59:6790/0 conn(0x7f4ef813f650 legacy=0x7f4ef81ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 >> v1:172.21.15.59:6789/0 conn(0x7f4ef81457f0 legacy=0x7f4ef813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 1 monclient: found mon.b 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef81cba50 con 0x7f4ef80a9d70 2024-03-29T11:19:50.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4effe8d640 5 monclient: authenticate success, global_id 4234 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4effe8d640 10 monclient: _renew_subs 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4effe8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4effe8d640 1 -- 172.21.15.59:0/726596005 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef81cef70 con 0x7f4ef80a9d70 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4effe8d640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4effe8d640 10 monclient: _renew_subs 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4effe8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4effe8d640 1 -- 172.21.15.59:0/726596005 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef8142760 con 0x7f4ef80a9d70 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4effe8d640 1 librados: init done 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ee8003990 con 0x7f4ef80a9d70 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4ee8003da0 con 0x7f4ef80a9d70 2024-03-29T11:19:50.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eee7fc640 10 monclient: dump: 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.487+0000 7f4eeffff640 4 set_mon_vals no callback set 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.489+0000 7f4effe8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:50.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.489+0000 7f4effe8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:50.477 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.489+0000 7f4effe8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:50.477 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.489+0000 7f4effe8d640 1 -- 172.21.15.59:0/726596005 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee0005080 con 0x7f4ef80a9d70 2024-03-29T11:19:50.477 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.490+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f4ee8003f00 con 0x7f4ef80a9d70 2024-03-29T11:19:50.478 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.491+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f4ee8084cf0 con 0x7f4ef80a9d70 2024-03-29T11:19:50.478 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.491+0000 7f4eee7fc640 10 client.4234.objecter ms_dispatch 0x7f4ef8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:50.478 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.491+0000 7f4eee7fc640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:50.478 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.491+0000 7f4eee7fc640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:50.478 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.491+0000 7f4eee7fc640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-29T11:19:50.478 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.491+0000 7f4efd428640 10 monclient: get_auth_request con 0x7f4edc06dfb0 auth_method 0 2024-03-29T11:19:50.478 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.491+0000 7f4eee7fc640 10 client.4234.objecter ms_handle_connect 0x7f4edc06dfb0 2024-03-29T11:19:50.478 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.496+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ee80505d0 con 0x7f4ef80a9d70 2024-03-29T11:19:50.478 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.496+0000 7f4eee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:50.478 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.496+0000 7f4eee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:50.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.904+0000 7f4effe8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:50.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.904+0000 7f4effe8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:50.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.904+0000 7f4effe8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:19:50.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.904+0000 7f4effe8d640 1 -- 172.21.15.59:0/726596005 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ee0005330 con 0x7f4ef80a9d70 2024-03-29T11:19:50.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.905+0000 7f4eee7fc640 1 -- 172.21.15.59:0/726596005 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f4ee8052500 con 0x7f4ef80a9d70 2024-03-29T11:19:50.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.905+0000 7f4eee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:50.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.905+0000 7f4eee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:50.882 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:19:50.883 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:50.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.909+0000 7f4effe8d640 10 librados: watch_flush enter 2024-03-29T11:19:50.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.909+0000 7f4effe8d640 10 librados: watch_flush exit 2024-03-29T11:19:50.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.909+0000 7f4effe8d640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-29T11:19:50.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.909+0000 7f4effe8d640 10 client.4234.objecter successfully canceled tick 2024-03-29T11:19:50.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.909+0000 7f4effe8d640 1 -- 172.21.15.59:0/726596005 >> v1:172.21.15.172:6816/2420742773 conn(0x7f4edc06dfb0 legacy=0x7f4edc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.909+0000 7f4effe8d640 10 monclient: shutdown 2024-03-29T11:19:50.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.909+0000 7f4effe8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:50.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.909+0000 7f4effe8d640 1 -- 172.21.15.59:0/726596005 >> v1:172.21.15.172:6789/0 conn(0x7f4ef80a9d70 legacy=0x7f4ef81cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.910+0000 7f4effe8d640 1 -- 172.21.15.59:0/726596005 shutdown_connections 2024-03-29T11:19:50.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.910+0000 7f4effe8d640 1 -- 172.21.15.59:0/726596005 >> 172.21.15.59:0/726596005 conn(0x7f4ef8066a00 msgr2=0x7f4ef80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:50.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.910+0000 7f4effe8d640 1 -- 172.21.15.59:0/726596005 shutdown_connections 2024-03-29T11:19:50.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.910+0000 7f4effe8d640 1 -- 172.21.15.59:0/726596005 wait complete. 2024-03-29T11:19:50.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:50.910+0000 7f4effe8d640 1 librados: shutdown 2024-03-29T11:19:50.902 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:57.709 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:57.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.880+0000 7f7b01532640 10 monclient: get_monmap_and_config 2024-03-29T11:19:57.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.880+0000 7f7b01532640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.880+0000 7f7b01532640 10 monclient: monmap: 2024-03-29T11:19:57.857 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:19:57.857 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:57.857 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:57.881500+0000 2024-03-29T11:19:57.857 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:57.881500+0000 2024-03-29T11:19:57.857 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.857 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:57.858 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:57.858 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:19:57.858 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:19:57.858 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:57.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.881+0000 7f7b01532640 1 Processor -- start 2024-03-29T11:19:57.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.881+0000 7f7b01532640 1 -- start start 2024-03-29T11:19:57.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.881+0000 7f7b01532640 10 monclient: init 2024-03-29T11:19:57.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.882+0000 7f7b01532640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.882+0000 7f7b01532640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:57.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.882+0000 7f7b01532640 10 monclient(hunting): picked mon.noname-c con 0x7f7afc1422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:57.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.883+0000 7f7b01532640 10 monclient(hunting): picked mon.noname-a con 0x7f7afc142690 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:57.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.883+0000 7f7b01532640 10 monclient(hunting): picked mon.noname-b con 0x7f7afc0b17d0 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:57.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.883+0000 7f7b01532640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc142c80 con 0x7f7afc142690 2024-03-29T11:19:57.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.883+0000 7f7b01532640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc0aae40 con 0x7f7afc1422b0 2024-03-29T11:19:57.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.883+0000 7f7b01532640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc13fbd0 con 0x7f7afc0b17d0 2024-03-29T11:19:57.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.883+0000 7f7b01532640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.883+0000 7f7b01532640 10 monclient(hunting): authenticate will time out at 847.996808s 2024-03-29T11:19:57.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.883+0000 7f7afa7fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7afc142690 0x7f7afc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57290/0 (socket says 172.21.15.59:57290) 2024-03-29T11:19:57.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.883+0000 7f7afa7fc640 1 -- 172.21.15.59:0/611811529 learned_addr learned my addr 172.21.15.59:0/611811529 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:19:57.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.883+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415346104 0 0) 0x7f7afc0aae40 con 0x7f7afc1422b0 2024-03-29T11:19:57.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:19:57.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae4003520 con 0x7f7afc1422b0 2024-03-29T11:19:57.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913222599 0 0) 0x7f7afc142c80 con 0x7f7afc142690 2024-03-29T11:19:57.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7afc0aae40 con 0x7f7afc142690 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980620193 0 0) 0x7f7afc13fbd0 con 0x7f7afc0b17d0 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7afc142c80 con 0x7f7afc0b17d0 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3072141832 0 0) 0x7f7ae4003520 con 0x7f7afc1422b0 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 >> v1:172.21.15.59:6790/0 conn(0x7f7afc0b17d0 legacy=0x7f7afc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 >> v1:172.21.15.59:6789/0 conn(0x7f7afc142690 legacy=0x7f7afc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:57.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc151940 con 0x7f7afc1422b0 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7b01532640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7af4001a10 con 0x7f7afc1422b0 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7af9ffb640 10 monclient: dump: 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:57.867 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.884+0000 7f7b01532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.885+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7af4001b70 con 0x7f7afc1422b0 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.885+0000 7f7af9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.885+0000 7f7af9ffb640 1 -- 172.21.15.59:0/611811529 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7af4001f40 con 0x7f7afc1422b0 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.885+0000 7f7af9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.885+0000 7f7af9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.885+0000 7f7af9ffb640 10 monclient: dump: 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.885+0000 7f7b01532640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.885+0000 7f7b01532640 4 set_mon_vals no callback set 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.886+0000 7f7b01532640 10 monclient: shutdown 2024-03-29T11:19:57.869 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.886+0000 7f7b01532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.886+0000 7f7b01532640 1 -- 172.21.15.59:0/611811529 >> v1:172.21.15.172:6789/0 conn(0x7f7afc1422b0 legacy=0x7f7afc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.886+0000 7f7b01532640 1 -- 172.21.15.59:0/611811529 shutdown_connections 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.886+0000 7f7b01532640 1 -- 172.21.15.59:0/611811529 >> 172.21.15.59:0/611811529 conn(0x7f7afc066a00 msgr2=0x7f7afc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.887+0000 7f7b01532640 1 -- 172.21.15.59:0/611811529 shutdown_connections 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.887+0000 7f7b01532640 1 -- 172.21.15.59:0/611811529 wait complete. 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.887+0000 7f7b01532640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.887+0000 7f7b01532640 10 monclient: monmap: 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:57.870 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:19:57.871 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:19:57.871 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:19:57.871 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:57.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.888+0000 7f7b01532640 1 librados: starting msgr at 2024-03-29T11:19:57.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.888+0000 7f7b01532640 1 librados: starting objecter 2024-03-29T11:19:57.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.888+0000 7f7b01532640 1 Processor -- start 2024-03-29T11:19:57.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.888+0000 7f7b01532640 1 -- start start 2024-03-29T11:19:57.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.888+0000 7f7b01532640 1 librados: setting wanted keys 2024-03-29T11:19:57.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.888+0000 7f7b01532640 1 librados: calling monclient init 2024-03-29T11:19:57.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.888+0000 7f7b01532640 10 monclient: init 2024-03-29T11:19:57.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.889+0000 7f7b01532640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.889+0000 7f7b01532640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:57.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.889+0000 7f7b01532640 10 monclient(hunting): picked mon.noname-b con 0x7f7afc0b17d0 addr v1:172.21.15.172:6789/0 2024-03-29T11:19:57.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.890+0000 7f7b01532640 10 monclient(hunting): picked mon.noname-c con 0x7f7afc1422b0 addr v1:172.21.15.59:6790/0 2024-03-29T11:19:57.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.890+0000 7f7b01532640 10 monclient(hunting): picked mon.noname-a con 0x7f7afc142690 addr v1:172.21.15.59:6789/0 2024-03-29T11:19:57.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.890+0000 7f7afa7fc640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f7afc1422b0 0x7f7afc12e860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:57984/0 (socket says 172.21.15.59:57984) 2024-03-29T11:19:57.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.890+0000 7f7afa7fc640 1 -- 172.21.15.59:0/4207025668 learned_addr learned my addr 172.21.15.59:0/4207025668 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:19:57.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.890+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc131e40 con 0x7f7afc142690 2024-03-29T11:19:57.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.890+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc12f680 con 0x7f7afc0b17d0 2024-03-29T11:19:57.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.890+0000 7f7ae37fe640 10 client.?.objecter ms_handle_connect 0x7f7afc142690 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7ae37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7ae37fe640 10 client.?.objecter ms_handle_connect 0x7f7afc1422b0 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7ae37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7ae37fe640 10 client.?.objecter ms_handle_connect 0x7f7afc0b17d0 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7ae37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655685039 0 0) 0x7f7afc131e40 con 0x7f7afc142690 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc06ea60 con 0x7f7afc1422b0 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7b01532640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7b01532640 10 monclient(hunting): authenticate will time out at 848.005394s 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7ae37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7ae37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7ae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7ae37fe640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:19:57.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.891+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae8004eb0 con 0x7f7afc142690 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746278063 0 0) 0x7f7afc12f680 con 0x7f7afc0b17d0 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7afc131e40 con 0x7f7afc0b17d0 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4046898678 0 0) 0x7f7ae8004eb0 con 0x7f7afc142690 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7afc12f680 con 0x7f7afc142690 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7aec001820 con 0x7f7afc142690 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): dump: 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:57.880 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246821978 0 0) 0x7f7afc06ea60 con 0x7f7afc1422b0 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae8004eb0 con 0x7f7afc1422b0 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3586325244 0 0) 0x7f7afc131e40 con 0x7f7afc0b17d0 2024-03-29T11:19:57.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.892+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7afc06ea60 con 0x7f7afc0b17d0 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7af4001e80 con 0x7f7afc0b17d0 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 10 monclient(hunting): dump: 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:57.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 194037261 0 0) 0x7f7afc12f680 con 0x7f7afc142690 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 >> v1:172.21.15.59:6790/0 conn(0x7f7afc1422b0 legacy=0x7f7afc12e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 >> v1:172.21.15.172:6789/0 conn(0x7f7afc0b17d0 legacy=0x7f7afc1dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 1 monclient: found mon.a 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc06ed40 con 0x7f7afc142690 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7ae37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7b01532640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7b01532640 10 monclient: _renew_subs 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7b01532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc06f020 con 0x7f7afc142690 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7b01532640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:57.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7b01532640 10 monclient: _renew_subs 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7b01532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc1324b0 con 0x7f7afc142690 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.893+0000 7f7b01532640 1 librados: init done 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.894+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7aec001d30 con 0x7f7afc142690 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.894+0000 7f7ae37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.894+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7aec0037b0 con 0x7f7afc142690 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.894+0000 7f7af8ff9640 4 set_mon_vals no callback set 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.894+0000 7f7ae37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.894+0000 7f7ae37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.894+0000 7f7ae37fe640 10 monclient: dump: 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.884 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.895+0000 7f7b01532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.895+0000 7f7b01532640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.895+0000 7f7b01532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.895+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7adc005080 con 0x7f7afc142690 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.897+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f7aec019780 con 0x7f7afc142690 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.898+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f7aec083710 con 0x7f7afc142690 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.898+0000 7f7afaffd640 10 monclient: get_auth_request con 0x7f7ae806de90 auth_method 0 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.898+0000 7f7ae37fe640 10 client.4245.objecter ms_dispatch 0x7f7afc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.898+0000 7f7ae37fe640 3 client.4245.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.898+0000 7f7ae37fe640 3 client.4245.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.898+0000 7f7ae37fe640 20 client.4245.objecter dump_active .. 0 homeless 2024-03-29T11:19:57.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.898+0000 7f7ae37fe640 10 client.4245.objecter ms_handle_connect 0x7f7ae806de90 2024-03-29T11:19:57.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.905+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7aec04ee20 con 0x7f7afc142690 2024-03-29T11:19:57.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.906+0000 7f7ae37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:57.906+0000 7f7ae37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:58.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.294+0000 7f7b01532640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.294+0000 7f7b01532640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.294+0000 7f7b01532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:19:58.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.294+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7adc005330 con 0x7f7afc142690 2024-03-29T11:19:58.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.295+0000 7f7ae37fe640 1 -- 172.21.15.59:0/4207025668 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f7aec050f20 con 0x7f7afc142690 2024-03-29T11:19:58.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.295+0000 7f7ae37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.295+0000 7f7ae37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:58.273 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:19:58.273 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:58.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.300+0000 7f7b01532640 10 librados: watch_flush enter 2024-03-29T11:19:58.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.300+0000 7f7b01532640 10 librados: watch_flush exit 2024-03-29T11:19:58.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.300+0000 7f7b01532640 20 client.4245.objecter shutdown clearing up homeless session... 2024-03-29T11:19:58.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.300+0000 7f7b01532640 10 client.4245.objecter successfully canceled tick 2024-03-29T11:19:58.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.300+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 >> v1:172.21.15.172:6816/2420742773 conn(0x7f7ae806de90 legacy=0x7f7ae8070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.300+0000 7f7b01532640 10 monclient: shutdown 2024-03-29T11:19:58.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.300+0000 7f7b01532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.300+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 >> v1:172.21.15.59:6789/0 conn(0x7f7afc142690 legacy=0x7f7afc12ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.300+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 shutdown_connections 2024-03-29T11:19:58.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.300+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 >> 172.21.15.59:0/4207025668 conn(0x7f7afc066a00 msgr2=0x7f7afc0b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.301+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 shutdown_connections 2024-03-29T11:19:58.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.301+0000 7f7b01532640 1 -- 172.21.15.59:0/4207025668 wait complete. 2024-03-29T11:19:58.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:19:58.301+0000 7f7b01532640 1 librados: shutdown 2024-03-29T11:19:58.292 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:05.098 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:05.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.266+0000 7fb028da3640 10 monclient: get_monmap_and_config 2024-03-29T11:20:05.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.266+0000 7fb028da3640 10 monclient: build_initial_monmap 2024-03-29T11:20:05.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.266+0000 7fb028da3640 10 monclient: monmap: 2024-03-29T11:20:05.244 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:05.244 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:05.244 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:20:05.267366+0000 2024-03-29T11:20:05.244 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:20:05.267366+0000 2024-03-29T11:20:05.244 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:05.244 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:05.244 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:05.244 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:20:05.244 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:20:05.245 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:05.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.267+0000 7fb028da3640 1 Processor -- start 2024-03-29T11:20:05.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.267+0000 7fb028da3640 1 -- start start 2024-03-29T11:20:05.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.267+0000 7fb028da3640 10 monclient: init 2024-03-29T11:20:05.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.268+0000 7fb028da3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:05.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.268+0000 7fb028da3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:05.250 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.271+0000 7fb028da3640 10 monclient(hunting): picked mon.noname-a con 0x7fb01c006890 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:05.250 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.271+0000 7fb028da3640 10 monclient(hunting): picked mon.noname-c con 0x7fb01c006c70 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:05.250 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.271+0000 7fb0237fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb01c006890 0x7fb01c0c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40683/0 (socket says 172.21.15.59:40683) 2024-03-29T11:20:05.250 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.271+0000 7fb0237fe640 1 -- 172.21.15.59:0/2435572573 learned_addr learned my addr 172.21.15.59:0/2435572573 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:05.250 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.271+0000 7fb028da3640 10 monclient(hunting): picked mon.noname-b con 0x7fb01c02b4c0 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:05.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.271+0000 7fb028da3640 1 -- 172.21.15.59:0/2435572573 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c02c830 con 0x7fb01c006890 2024-03-29T11:20:05.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.271+0000 7fb028da3640 1 -- 172.21.15.59:0/2435572573 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c02afa0 con 0x7fb01c006c70 2024-03-29T11:20:05.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.271+0000 7fb028da3640 1 -- 172.21.15.59:0/2435572573 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c02aba0 con 0x7fb01c02b4c0 2024-03-29T11:20:05.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.271+0000 7fb028da3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:05.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.271+0000 7fb028da3640 10 monclient(hunting): authenticate will time out at 855.385443s 2024-03-29T11:20:05.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 1 -- 172.21.15.59:0/2435572573 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366335418 0 0) 0x7fb01c02c830 con 0x7fb01c006890 2024-03-29T11:20:05.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:05.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:05.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:05.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:20:05.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 1 -- 172.21.15.59:0/2435572573 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb014003520 con 0x7fb01c006890 2024-03-29T11:20:05.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 1 -- 172.21.15.59:0/2435572573 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730051730 0 0) 0x7fb01c02afa0 con 0x7fb01c006c70 2024-03-29T11:20:05.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:05.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:05.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:05.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:20:05.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.272+0000 7fb0227fc640 1 -- 172.21.15.59:0/2435572573 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c02c830 con 0x7fb01c006c70 2024-03-29T11:20:05.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.273+0000 7fb0227fc640 1 -- 172.21.15.59:0/2435572573 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3236624637 0 0) 0x7fb014003520 con 0x7fb01c006890 2024-03-29T11:20:05.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.273+0000 7fb0227fc640 1 -- 172.21.15.59:0/2435572573 >> v1:172.21.15.59:6790/0 conn(0x7fb01c02b4c0 legacy=0x7fb01c0cf0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.274+0000 7fb0227fc640 1 -- 172.21.15.59:0/2435572573 >> v1:172.21.15.172:6789/0 conn(0x7fb01c006c70 legacy=0x7fb01c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.274+0000 7fb0227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.274+0000 7fb0227fc640 1 monclient: found mon.noname-a 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.274+0000 7fb0227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.274+0000 7fb0227fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.274+0000 7fb0227fc640 1 -- 172.21.15.59:0/2435572573 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c02ad30 con 0x7fb01c006890 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb0227fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb0227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb0227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb0227fc640 1 -- 172.21.15.59:0/2435572573 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb018000ed0 con 0x7fb01c006890 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb028da3640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:20:05.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb028da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:05.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb0227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb0227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:05.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb0227fc640 10 monclient: dump: 2024-03-29T11:20:05.260 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:05.260 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:05.260 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:05.260 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:05.260 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.260 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:05.260 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:05.261 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:05.261 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:05.261 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:05.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb028da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:05.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb0227fc640 1 -- 172.21.15.59:0/2435572573 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0180010f0 con 0x7fb01c006890 2024-03-29T11:20:05.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb0227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:05.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb0227fc640 1 -- 172.21.15.59:0/2435572573 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb018001400 con 0x7fb01c006890 2024-03-29T11:20:05.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb028da3640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:05.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb028da3640 4 set_mon_vals no callback set 2024-03-29T11:20:05.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.275+0000 7fb0227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.276+0000 7fb0227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:05.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.276+0000 7fb0227fc640 10 monclient: dump: 2024-03-29T11:20:05.262 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:05.262 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:05.262 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:05.262 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.276+0000 7fb028da3640 10 monclient: shutdown 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.276+0000 7fb028da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.276+0000 7fb028da3640 1 -- 172.21.15.59:0/2435572573 >> v1:172.21.15.59:6789/0 conn(0x7fb01c006890 legacy=0x7fb01c0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.276+0000 7fb028da3640 1 -- 172.21.15.59:0/2435572573 shutdown_connections 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.276+0000 7fb028da3640 1 -- 172.21.15.59:0/2435572573 >> 172.21.15.59:0/2435572573 conn(0x7fb01c0058f0 msgr2=0x7fb01c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.277+0000 7fb028da3640 1 -- 172.21.15.59:0/2435572573 shutdown_connections 2024-03-29T11:20:05.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.277+0000 7fb028da3640 1 -- 172.21.15.59:0/2435572573 wait complete. 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.278+0000 7fb028da3640 10 monclient: build_initial_monmap 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.278+0000 7fb028da3640 10 monclient: monmap: 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:05.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.279+0000 7fb028da3640 1 librados: starting msgr at 2024-03-29T11:20:05.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.279+0000 7fb028da3640 1 librados: starting objecter 2024-03-29T11:20:05.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.279+0000 7fb028da3640 1 Processor -- start 2024-03-29T11:20:05.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.279+0000 7fb028da3640 1 -- start start 2024-03-29T11:20:05.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.279+0000 7fb028da3640 1 librados: setting wanted keys 2024-03-29T11:20:05.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.279+0000 7fb028da3640 1 librados: calling monclient init 2024-03-29T11:20:05.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.279+0000 7fb028da3640 10 monclient: init 2024-03-29T11:20:05.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.280+0000 7fb028da3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:05.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.280+0000 7fb028da3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:05.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.284+0000 7fb028da3640 10 monclient(hunting): picked mon.noname-a con 0x7fb01c006890 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:05.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.284+0000 7fb028da3640 10 monclient(hunting): picked mon.noname-b con 0x7fb01c02b4c0 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:05.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.284+0000 7fb028da3640 10 monclient(hunting): picked mon.noname-c con 0x7fb01c0b0e10 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:05.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.284+0000 7fb028da3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c15a8c0 con 0x7fb01c006890 2024-03-29T11:20:05.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.284+0000 7fb028da3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c0b2870 con 0x7fb01c02b4c0 2024-03-29T11:20:05.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.284+0000 7fb028da3640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c0b2b50 con 0x7fb01c0b0e10 2024-03-29T11:20:05.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.284+0000 7fb028da3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:05.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.284+0000 7fb028da3640 10 monclient(hunting): authenticate will time out at 855.398532s 2024-03-29T11:20:05.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.285+0000 7fb0237fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb01c006890 0x7fb01c15e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40833/0 (socket says 172.21.15.59:40833) 2024-03-29T11:20:05.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.285+0000 7fb0237fe640 1 -- 172.21.15.59:0/3091071946 learned_addr learned my addr 172.21.15.59:0/3091071946 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:05.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.285+0000 7fb00ffff640 10 client.?.objecter ms_handle_connect 0x7fb01c006890 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.285+0000 7fb00ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.285+0000 7fb00ffff640 10 client.?.objecter ms_handle_connect 0x7fb01c02b4c0 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.285+0000 7fb00ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.285+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860067377 0 0) 0x7fb01c15a8c0 con 0x7fb01c006890 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.285+0000 7fb00ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.285+0000 7fb00ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.285+0000 7fb00ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb008004eb0 con 0x7fb01c006890 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657014943 0 0) 0x7fb01c0b2870 con 0x7fb01c02b4c0 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c15a8c0 con 0x7fb01c02b4c0 2024-03-29T11:20:05.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3319508234 0 0) 0x7fb008004eb0 con 0x7fb01c006890 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb01c0b2870 con 0x7fb01c006890 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb018001820 con 0x7fb01c006890 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.286+0000 7fb00ffff640 10 monclient(hunting): dump: 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:05.276 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.287+0000 7fb00ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.287+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1212451935 0 0) 0x7fb01c0b2870 con 0x7fb01c006890 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.287+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 >> v1:172.21.15.59:6790/0 conn(0x7fb01c0b0e10 legacy=0x7fb01c166100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 >> v1:172.21.15.172:6789/0 conn(0x7fb01c02b4c0 legacy=0x7fb01c15ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb00ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb00ffff640 1 monclient: found mon.a 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb00ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb00ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c031a90 con 0x7fb01c006890 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb00ffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb00ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb00ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb028da3640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb028da3640 10 monclient: _renew_subs 2024-03-29T11:20:05.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.288+0000 7fb028da3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.289+0000 7fb028da3640 1 -- 172.21.15.59:0/3091071946 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c0b8770 con 0x7fb01c006890 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.289+0000 7fb028da3640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.289+0000 7fb028da3640 10 monclient: _renew_subs 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.289+0000 7fb028da3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.289+0000 7fb028da3640 1 -- 172.21.15.59:0/3091071946 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01c15ea20 con 0x7fb01c006890 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.289+0000 7fb028da3640 1 librados: init done 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.289+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb018001cf0 con 0x7fb01c006890 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.289+0000 7fb00ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.290+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb018004db0 con 0x7fb01c006890 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.290+0000 7fb021ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.290+0000 7fb00ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.290+0000 7fb00ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.290+0000 7fb00ffff640 10 monclient: dump: 2024-03-29T11:20:05.278 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:05.279 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:05.279 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:05.279 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:05.279 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.279 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:05.279 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:05.279 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:05.279 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:05.279 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:05.279 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.290+0000 7fb028da3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:05.279 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.291+0000 7fb028da3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:05.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.291+0000 7fb028da3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:05.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.291+0000 7fb028da3640 1 -- 172.21.15.59:0/3091071946 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb004005080 con 0x7fb01c006890 2024-03-29T11:20:05.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.293+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fb01801ad80 con 0x7fb01c006890 2024-03-29T11:20:05.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.294+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fb018084b70 con 0x7fb01c006890 2024-03-29T11:20:05.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.294+0000 7fb00ffff640 10 client.4251.objecter ms_dispatch 0x7fb024000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:05.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.294+0000 7fb00ffff640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:05.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.295+0000 7fb022ffd640 10 monclient: get_auth_request con 0x7fb00806db60 auth_method 0 2024-03-29T11:20:05.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.295+0000 7fb00ffff640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:05.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.295+0000 7fb00ffff640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:20:05.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.295+0000 7fb00ffff640 10 client.4251.objecter ms_handle_connect 0x7fb00806db60 2024-03-29T11:20:05.281 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.300+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb018050450 con 0x7fb01c006890 2024-03-29T11:20:05.281 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.300+0000 7fb00ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:05.281 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.300+0000 7fb00ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:05.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.632+0000 7fb028da3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:05.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.632+0000 7fb028da3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:05.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.632+0000 7fb028da3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:05.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.632+0000 7fb028da3640 1 -- 172.21.15.59:0/3091071946 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb004005330 con 0x7fb01c006890 2024-03-29T11:20:05.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.633+0000 7fb00ffff640 1 -- 172.21.15.59:0/3091071946 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fb018052380 con 0x7fb01c006890 2024-03-29T11:20:05.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.633+0000 7fb00ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:05.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.633+0000 7fb00ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:05.610 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:20:05.610 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:05.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.638+0000 7fb028da3640 10 librados: watch_flush enter 2024-03-29T11:20:05.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.638+0000 7fb028da3640 10 librados: watch_flush exit 2024-03-29T11:20:05.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.638+0000 7fb028da3640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:20:05.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.638+0000 7fb028da3640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:20:05.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.638+0000 7fb028da3640 1 -- 172.21.15.59:0/3091071946 >> v1:172.21.15.172:6816/2420742773 conn(0x7fb00806db60 legacy=0x7fb008070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.638+0000 7fb028da3640 10 monclient: shutdown 2024-03-29T11:20:05.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.638+0000 7fb028da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:05.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.638+0000 7fb028da3640 1 -- 172.21.15.59:0/3091071946 >> v1:172.21.15.59:6789/0 conn(0x7fb01c006890 legacy=0x7fb01c15e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.639+0000 7fb028da3640 1 -- 172.21.15.59:0/3091071946 shutdown_connections 2024-03-29T11:20:05.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.639+0000 7fb028da3640 1 -- 172.21.15.59:0/3091071946 >> 172.21.15.59:0/3091071946 conn(0x7fb01c0058f0 msgr2=0x7fb01c005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:05.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.639+0000 7fb028da3640 1 -- 172.21.15.59:0/3091071946 shutdown_connections 2024-03-29T11:20:05.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.639+0000 7fb028da3640 1 -- 172.21.15.59:0/3091071946 wait complete. 2024-03-29T11:20:05.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:05.639+0000 7fb028da3640 1 librados: shutdown 2024-03-29T11:20:05.633 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:12.441 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:12.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.608+0000 7faf1ca0f640 10 monclient: get_monmap_and_config 2024-03-29T11:20:12.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.608+0000 7faf1ca0f640 10 monclient: build_initial_monmap 2024-03-29T11:20:12.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.608+0000 7faf1ca0f640 10 monclient: monmap: 2024-03-29T11:20:12.586 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:12.586 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:12.586 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:20:12.609248+0000 2024-03-29T11:20:12.587 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:20:12.609248+0000 2024-03-29T11:20:12.587 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:12.587 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:12.587 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:12.587 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:20:12.587 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:20:12.588 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:12.593 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.609+0000 7faf1ca0f640 1 Processor -- start 2024-03-29T11:20:12.593 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.609+0000 7faf1ca0f640 1 -- start start 2024-03-29T11:20:12.593 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.609+0000 7faf1ca0f640 10 monclient: init 2024-03-29T11:20:12.593 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.610+0000 7faf1ca0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:12.593 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.610+0000 7faf1ca0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:12.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.610+0000 7faf1ca0f640 10 monclient(hunting): picked mon.noname-b con 0x7faf1806d9d0 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:12.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.610+0000 7faf1ca0f640 10 monclient(hunting): picked mon.noname-c con 0x7faf180b0860 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:12.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.610+0000 7faf1ca0f640 10 monclient(hunting): picked mon.noname-a con 0x7faf180af8e0 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:12.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.610+0000 7faf1ca0f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf180aed10 con 0x7faf180af8e0 2024-03-29T11:20:12.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.610+0000 7faf1ca0f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf180afd50 con 0x7faf180b0860 2024-03-29T11:20:12.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.610+0000 7faf1ca0f640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf180ab1b0 con 0x7faf1806d9d0 2024-03-29T11:20:12.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.610+0000 7faf1ca0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:12.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.610+0000 7faf1ca0f640 10 monclient(hunting): authenticate will time out at 862.724497s 2024-03-29T11:20:12.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.611+0000 7faf15d9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7faf180b0860 0x7faf18146a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:42066/0 (socket says 172.21.15.59:42066) 2024-03-29T11:20:12.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.611+0000 7faf15d9b640 1 -- 172.21.15.59:0/3256366318 learned_addr learned my addr 172.21.15.59:0/3256366318 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:12.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.611+0000 7faf1559a640 1 -- 172.21.15.59:0/3256366318 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353816449 0 0) 0x7faf180afd50 con 0x7faf180b0860 2024-03-29T11:20:12.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.611+0000 7faf1559a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:12.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.611+0000 7faf1559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:12.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.611+0000 7faf1559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:12.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.611+0000 7faf1559a640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:20:12.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.611+0000 7faf1559a640 1 -- 172.21.15.59:0/3256366318 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf00003520 con 0x7faf180b0860 2024-03-29T11:20:12.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.612+0000 7faf1559a640 1 -- 172.21.15.59:0/3256366318 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3198360031 0 0) 0x7faf00003520 con 0x7faf180b0860 2024-03-29T11:20:12.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.612+0000 7faf1559a640 1 -- 172.21.15.59:0/3256366318 >> v1:172.21.15.59:6790/0 conn(0x7faf1806d9d0 legacy=0x7faf180af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:12.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.614+0000 7faf1559a640 1 -- 172.21.15.59:0/3256366318 >> v1:172.21.15.59:6789/0 conn(0x7faf180af8e0 legacy=0x7faf1814a180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:12.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:12.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 1 monclient: found mon.noname-c 2024-03-29T11:20:12.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:12.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:20:12.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 1 -- 172.21.15.59:0/3256366318 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf180ab490 con 0x7faf180b0860 2024-03-29T11:20:12.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 10 monclient: _finish_auth 0 2024-03-29T11:20:12.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:12.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:12.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 1 -- 172.21.15.59:0/3256366318 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7faf08001690 con 0x7faf180b0860 2024-03-29T11:20:12.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:12.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:12.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1559a640 10 monclient: dump: 2024-03-29T11:20:12.599 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:12.599 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:12.599 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:12.599 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:12.600 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:12.600 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:12.600 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:12.600 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:12.600 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:12.601 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:12.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1ca0f640 5 monclient: authenticate success, global_id 4249 2024-03-29T11:20:12.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.615+0000 7faf1ca0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:12.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.616+0000 7faf1559a640 1 -- 172.21.15.59:0/3256366318 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf08001860 con 0x7faf180b0860 2024-03-29T11:20:12.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.616+0000 7faf1559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:12.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.616+0000 7faf1559a640 1 -- 172.21.15.59:0/3256366318 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7faf08001cb0 con 0x7faf180b0860 2024-03-29T11:20:12.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.616+0000 7faf1ca0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:12.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.616+0000 7faf1ca0f640 4 set_mon_vals no callback set 2024-03-29T11:20:12.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.616+0000 7faf1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:12.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.616+0000 7faf1559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:12.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.616+0000 7faf1559a640 10 monclient: dump: 2024-03-29T11:20:12.602 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:12.603 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:12.603 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:12.603 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:12.603 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:12.603 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:12.603 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:12.604 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:12.604 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:12.604 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:12.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.617+0000 7faf1ca0f640 10 monclient: shutdown 2024-03-29T11:20:12.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.617+0000 7faf1ca0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:12.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.617+0000 7faf1ca0f640 1 -- 172.21.15.59:0/3256366318 >> v1:172.21.15.172:6789/0 conn(0x7faf180b0860 legacy=0x7faf18146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.617+0000 7faf1ca0f640 1 -- 172.21.15.59:0/3256366318 shutdown_connections 2024-03-29T11:20:12.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.617+0000 7faf1ca0f640 1 -- 172.21.15.59:0/3256366318 >> 172.21.15.59:0/3256366318 conn(0x7faf1806b660 msgr2=0x7faf1806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:12.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.617+0000 7faf1ca0f640 1 -- 172.21.15.59:0/3256366318 shutdown_connections 2024-03-29T11:20:12.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.617+0000 7faf1ca0f640 1 -- 172.21.15.59:0/3256366318 wait complete. 2024-03-29T11:20:12.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.618+0000 7faf1ca0f640 10 monclient: build_initial_monmap 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.618+0000 7faf1ca0f640 10 monclient: monmap: 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.619+0000 7faf1ca0f640 1 librados: starting msgr at 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.619+0000 7faf1ca0f640 1 librados: starting objecter 2024-03-29T11:20:12.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.619+0000 7faf1ca0f640 1 Processor -- start 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.619+0000 7faf1ca0f640 1 -- start start 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.619+0000 7faf1ca0f640 1 librados: setting wanted keys 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.619+0000 7faf1ca0f640 1 librados: calling monclient init 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.619+0000 7faf1ca0f640 10 monclient: init 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.620+0000 7faf1ca0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.620+0000 7faf1ca0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.620+0000 7faf1ca0f640 10 monclient(hunting): picked mon.noname-a con 0x7faf1806d9d0 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.621+0000 7faf1ca0f640 10 monclient(hunting): picked mon.noname-c con 0x7faf180af8e0 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.621+0000 7faf1ca0f640 10 monclient(hunting): picked mon.noname-b con 0x7faf180b0860 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.621+0000 7faf1ca0f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf18138330 con 0x7faf1806d9d0 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.621+0000 7faf1ca0f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf1806f4d0 con 0x7faf180b0860 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.621+0000 7faf1ca0f640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf1806e270 con 0x7faf180af8e0 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.621+0000 7faf1ca0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:12.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.621+0000 7faf1ca0f640 10 monclient(hunting): authenticate will time out at 862.735183s 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.623+0000 7faf16d9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7faf180b0860 0x7faf181e1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:42246/0 (socket says 172.21.15.59:42246) 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.623+0000 7faf15d9b640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7faf180af8e0 0x7faf181d7bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:46965/0 (socket says 172.21.15.59:46965) 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.623+0000 7faf15d9b640 1 -- 172.21.15.59:0/1354263452 learned_addr learned my addr 172.21.15.59:0/1354263452 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.623+0000 7faf06ffd640 10 client.?.objecter ms_handle_connect 0x7faf180af8e0 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 10 client.?.objecter ms_handle_connect 0x7faf1806d9d0 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344347117 0 0) 0x7faf1806e270 con 0x7faf180af8e0 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:20:12.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef8004eb0 con 0x7faf180af8e0 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110715824 0 0) 0x7faf18138330 con 0x7faf1806d9d0 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf1806e270 con 0x7faf1806d9d0 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 403469890 0 0) 0x7faef8004eb0 con 0x7faf180af8e0 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf18138330 con 0x7faf180af8e0 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.624+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7faf08001c30 con 0x7faf180af8e0 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:12.622 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4022427059 0 0) 0x7faf1806e270 con 0x7faf1806d9d0 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faef8004eb0 con 0x7faf1806d9d0 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 10 client.?.objecter ms_handle_connect 0x7faf180b0860 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:12.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2356286144 0 0) 0x7faf18138330 con 0x7faf180af8e0 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 >> v1:172.21.15.172:6789/0 conn(0x7faf180b0860 legacy=0x7faf181e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 >> v1:172.21.15.59:6789/0 conn(0x7faf1806d9d0 legacy=0x7faf181d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.625+0000 7faf06ffd640 1 monclient: found mon.c 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf06ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf181387c0 con 0x7faf180af8e0 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf06ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf06ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf1ca0f640 5 monclient: authenticate success, global_id 4241 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf1ca0f640 10 monclient: _renew_subs 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf1ca0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:12.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf1ca0f640 1 -- 172.21.15.59:0/1354263452 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf18135580 con 0x7faf180af8e0 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf1ca0f640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf1ca0f640 10 monclient: _renew_subs 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf1ca0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf1ca0f640 1 -- 172.21.15.59:0/1354263452 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1806ddb0 con 0x7faf180af8e0 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf1ca0f640 1 librados: init done 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf08004b00 con 0x7faf180af8e0 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf06ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7faf08004e90 con 0x7faf180af8e0 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf14d99640 4 set_mon_vals no callback set 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.626+0000 7faf06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.627+0000 7faf06ffd640 10 monclient: dump: 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:12.625 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.627+0000 7faf1ca0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.627+0000 7faf1ca0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.627+0000 7faf1ca0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.627+0000 7faf1ca0f640 1 -- 172.21.15.59:0/1354263452 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1806f310 con 0x7faf180af8e0 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.628+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7faf0801ae60 con 0x7faf180af8e0 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.628+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7faf08084ab0 con 0x7faf180af8e0 2024-03-29T11:20:12.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.629+0000 7faf06ffd640 10 client.4241.objecter ms_dispatch 0x7faf18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:12.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.629+0000 7faf06ffd640 3 client.4241.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:12.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.629+0000 7faf06ffd640 3 client.4241.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:12.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.629+0000 7faf1659c640 10 monclient: get_auth_request con 0x7faef806dd70 auth_method 0 2024-03-29T11:20:12.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.629+0000 7faf06ffd640 20 client.4241.objecter dump_active .. 0 homeless 2024-03-29T11:20:12.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.629+0000 7faf06ffd640 10 client.4241.objecter ms_handle_connect 0x7faef806dd70 2024-03-29T11:20:12.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.637+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf08050390 con 0x7faf180af8e0 2024-03-29T11:20:12.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.637+0000 7faf06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:12.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.637+0000 7faf06ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:12.943 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.966+0000 7faf1ca0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:12.943 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.966+0000 7faf1ca0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:12.943 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.966+0000 7faf1ca0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:12.943 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.966+0000 7faf1ca0f640 1 -- 172.21.15.59:0/1354263452 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf18075f10 con 0x7faf180af8e0 2024-03-29T11:20:12.944 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.967+0000 7faf06ffd640 1 -- 172.21.15.59:0/1354263452 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7faf080522c0 con 0x7faf180af8e0 2024-03-29T11:20:12.944 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.967+0000 7faf06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:12.944 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.967+0000 7faf06ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:12.944 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:20:12.944 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:12.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.971+0000 7faf1ca0f640 10 librados: watch_flush enter 2024-03-29T11:20:12.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.972+0000 7faf1ca0f640 10 librados: watch_flush exit 2024-03-29T11:20:12.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.972+0000 7faf1ca0f640 20 client.4241.objecter shutdown clearing up homeless session... 2024-03-29T11:20:12.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.972+0000 7faf1ca0f640 10 client.4241.objecter successfully canceled tick 2024-03-29T11:20:12.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.972+0000 7faf1ca0f640 1 -- 172.21.15.59:0/1354263452 >> v1:172.21.15.172:6816/2420742773 conn(0x7faef806dd70 legacy=0x7faef8070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.972+0000 7faf1ca0f640 10 monclient: shutdown 2024-03-29T11:20:12.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.972+0000 7faf1ca0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:12.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.972+0000 7faf1ca0f640 1 -- 172.21.15.59:0/1354263452 >> v1:172.21.15.59:6790/0 conn(0x7faf180af8e0 legacy=0x7faf181d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.972+0000 7faf1ca0f640 1 -- 172.21.15.59:0/1354263452 shutdown_connections 2024-03-29T11:20:12.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.972+0000 7faf1ca0f640 1 -- 172.21.15.59:0/1354263452 >> 172.21.15.59:0/1354263452 conn(0x7faf1806b660 msgr2=0x7faf1813ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:12.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.972+0000 7faf1ca0f640 1 -- 172.21.15.59:0/1354263452 shutdown_connections 2024-03-29T11:20:12.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.972+0000 7faf1ca0f640 1 -- 172.21.15.59:0/1354263452 wait complete. 2024-03-29T11:20:12.950 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:12.973+0000 7faf1ca0f640 1 librados: shutdown 2024-03-29T11:20:12.966 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:19.773 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:19.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.942+0000 7f06161a3640 10 monclient: get_monmap_and_config 2024-03-29T11:20:19.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.942+0000 7f06161a3640 10 monclient: build_initial_monmap 2024-03-29T11:20:19.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.942+0000 7f06161a3640 10 monclient: monmap: 2024-03-29T11:20:19.920 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:19.920 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:19.920 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:20:19.944000+0000 2024-03-29T11:20:19.920 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:20:19.944000+0000 2024-03-29T11:20:19.920 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:19.920 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:19.920 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:19.920 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:20:19.921 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:20:19.921 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:19.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.943+0000 7f06161a3640 1 Processor -- start 2024-03-29T11:20:19.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.944+0000 7f06161a3640 1 -- start start 2024-03-29T11:20:19.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.944+0000 7f06161a3640 10 monclient: init 2024-03-29T11:20:19.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.945+0000 7f06161a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:19.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.945+0000 7f06161a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:19.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.945+0000 7f06161a3640 10 monclient(hunting): picked mon.noname-b con 0x7f06100b0860 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:19.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.945+0000 7f06161a3640 10 monclient(hunting): picked mon.noname-a con 0x7f061013ef30 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:19.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.945+0000 7f06161a3640 10 monclient(hunting): picked mon.noname-c con 0x7f06100af8e0 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:19.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.945+0000 7f06161a3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06100b0c40 con 0x7f061013ef30 2024-03-29T11:20:19.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.945+0000 7f06161a3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06100aed10 con 0x7f06100af8e0 2024-03-29T11:20:19.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.945+0000 7f06161a3640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06100afd50 con 0x7f06100b0860 2024-03-29T11:20:19.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.945+0000 7f06161a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:19.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.945+0000 7f06161a3640 10 monclient(hunting): authenticate will time out at 870.060491s 2024-03-29T11:20:19.926 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.949+0000 7f060ffff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f06100af8e0 0x7f0610149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:53863/0 (socket says 172.21.15.59:53863) 2024-03-29T11:20:19.926 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.949+0000 7f060ffff640 1 -- 172.21.15.59:0/2851999125 learned_addr learned my addr 172.21.15.59:0/2851999125 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:19.927 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.950+0000 7f060e7fc640 1 -- 172.21.15.59:0/2851999125 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201500083 0 0) 0x7f06100aed10 con 0x7f06100af8e0 2024-03-29T11:20:19.927 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.950+0000 7f060e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:19.927 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.950+0000 7f060e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:19.928 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.950+0000 7f060e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:19.928 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.950+0000 7f060e7fc640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:20:19.928 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.950+0000 7f060e7fc640 1 -- 172.21.15.59:0/2851999125 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05f8003520 con 0x7f06100af8e0 2024-03-29T11:20:19.928 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.951+0000 7f060e7fc640 1 -- 172.21.15.59:0/2851999125 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2381169764 0 0) 0x7f05f8003520 con 0x7f06100af8e0 2024-03-29T11:20:19.928 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.951+0000 7f060e7fc640 1 -- 172.21.15.59:0/2851999125 >> v1:172.21.15.59:6790/0 conn(0x7f06100b0860 legacy=0x7f061013e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:19.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.951+0000 7f060e7fc640 1 -- 172.21.15.59:0/2851999125 >> v1:172.21.15.59:6789/0 conn(0x7f061013ef30 legacy=0x7f06100af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:19.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:19.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:20:19.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:19.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:20:19.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 1 -- 172.21.15.59:0/2851999125 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06100ab1b0 con 0x7f06100af8e0 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 1 -- 172.21.15.59:0/2851999125 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0608001a10 con 0x7f06100af8e0 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 10 monclient: dump: 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:19.933 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:19.934 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:19.934 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:19.934 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:19.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 1 -- 172.21.15.59:0/2851999125 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0608001be0 con 0x7f06100af8e0 2024-03-29T11:20:19.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:19.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 1 -- 172.21.15.59:0/2851999125 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0608002030 con 0x7f06100af8e0 2024-03-29T11:20:19.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:19.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.955+0000 7f060e7fc640 10 monclient: dump: 2024-03-29T11:20:19.935 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:19.935 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.956+0000 7f06161a3640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.956+0000 7f06161a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.956+0000 7f06161a3640 4 set_mon_vals no callback set 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.957+0000 7f06161a3640 10 monclient: shutdown 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.957+0000 7f06161a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:19.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.957+0000 7f06161a3640 1 -- 172.21.15.59:0/2851999125 >> v1:172.21.15.172:6789/0 conn(0x7f06100af8e0 legacy=0x7f0610149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.937 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.957+0000 7f06161a3640 1 -- 172.21.15.59:0/2851999125 shutdown_connections 2024-03-29T11:20:19.937 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.957+0000 7f06161a3640 1 -- 172.21.15.59:0/2851999125 >> 172.21.15.59:0/2851999125 conn(0x7f061006b660 msgr2=0x7f061006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:19.937 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.959+0000 7f06161a3640 1 -- 172.21.15.59:0/2851999125 shutdown_connections 2024-03-29T11:20:19.938 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.959+0000 7f06161a3640 1 -- 172.21.15.59:0/2851999125 wait complete. 2024-03-29T11:20:19.938 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.960+0000 7f06161a3640 10 monclient: build_initial_monmap 2024-03-29T11:20:19.938 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.960+0000 7f06161a3640 10 monclient: monmap: 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.960+0000 7f06161a3640 1 librados: starting msgr at 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.960+0000 7f06161a3640 1 librados: starting objecter 2024-03-29T11:20:19.939 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.961+0000 7f06161a3640 1 Processor -- start 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.961+0000 7f06161a3640 1 -- start start 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.961+0000 7f06161a3640 1 librados: setting wanted keys 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.961+0000 7f06161a3640 1 librados: calling monclient init 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.961+0000 7f06161a3640 10 monclient: init 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.962+0000 7f06161a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.962+0000 7f06161a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.962+0000 7f06161a3640 10 monclient(hunting): picked mon.noname-a con 0x7f06100af8e0 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.962+0000 7f06161a3640 10 monclient(hunting): picked mon.noname-b con 0x7f06100b0860 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.962+0000 7f06161a3640 10 monclient(hunting): picked mon.noname-c con 0x7f061013ef30 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.962+0000 7f06161a3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06101386f0 con 0x7f06100af8e0 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.962+0000 7f06161a3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061006ea70 con 0x7f06100b0860 2024-03-29T11:20:19.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.962+0000 7f06161a3640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061006ed50 con 0x7f061013ef30 2024-03-29T11:20:19.942 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.962+0000 7f06161a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:19.942 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.962+0000 7f06161a3640 10 monclient(hunting): authenticate will time out at 870.077321s 2024-03-29T11:20:19.944 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.967+0000 7f060f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f06100af8e0 0x7f06101dec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:43253/0 (socket says 172.21.15.59:43253) 2024-03-29T11:20:19.944 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.967+0000 7f060f7fe640 1 -- 172.21.15.59:0/407001027 learned_addr learned my addr 172.21.15.59:0/407001027 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:19.944 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.967+0000 7f05f7fff640 10 client.?.objecter ms_handle_connect 0x7f06100af8e0 2024-03-29T11:20:19.944 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.967+0000 7f05f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:19.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.968+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536819651 0 0) 0x7f06101386f0 con 0x7f06100af8e0 2024-03-29T11:20:19.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.968+0000 7f05f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:19.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.968+0000 7f05f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:19.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.968+0000 7f05f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:19.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.968+0000 7f05f7fff640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:20:19.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.968+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05fc004eb0 con 0x7f06100af8e0 2024-03-29T11:20:19.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 client.?.objecter ms_handle_connect 0x7f061013ef30 2024-03-29T11:20:19.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263239036 0 0) 0x7f061006ed50 con 0x7f061013ef30 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06101386f0 con 0x7f061013ef30 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2573246581 0 0) 0x7f06101386f0 con 0x7f061013ef30 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f061006ed50 con 0x7f061013ef30 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f06080013b0 con 0x7f061013ef30 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 monclient(hunting): dump: 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:19.949 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 53862140 0 0) 0x7f05fc004eb0 con 0x7f06100af8e0 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06101386f0 con 0x7f06100af8e0 2024-03-29T11:20:19.950 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0604001960 con 0x7f06100af8e0 2024-03-29T11:20:19.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:19.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 10 monclient(hunting): dump: 2024-03-29T11:20:19.954 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:19.954 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:19.954 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:19.954 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:19.954 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.954 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:19.954 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:19.954 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3609970681 0 0) 0x7f061006ed50 con 0x7f061013ef30 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.969+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 >> v1:172.21.15.172:6789/0 conn(0x7f06100b0860 legacy=0x7f0610136590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 >> v1:172.21.15.59:6789/0 conn(0x7f06100af8e0 legacy=0x7f06101dec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 1 monclient: found mon.c 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061006f030 con 0x7f061013ef30 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:19.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0608001f70 con 0x7f061013ef30 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f06080050e0 con 0x7f061013ef30 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f05f7fff640 10 monclient: dump: 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:19.956 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f060dffb640 4 set_mon_vals no callback set 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f06161a3640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f06161a3640 10 monclient: _renew_subs 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f06161a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.970+0000 7f06161a3640 1 -- 172.21.15.59:0/407001027 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061006dd20 con 0x7f061013ef30 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.971+0000 7f06161a3640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.971+0000 7f06161a3640 10 monclient: _renew_subs 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.971+0000 7f06161a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.971+0000 7f06161a3640 1 -- 172.21.15.59:0/407001027 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0610137ad0 con 0x7f061013ef30 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.971+0000 7f06161a3640 1 librados: init done 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.973+0000 7f06161a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.973+0000 7f06161a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:19.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.973+0000 7f06161a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:19.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.973+0000 7f06161a3640 1 -- 172.21.15.59:0/407001027 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0610138550 con 0x7f061013ef30 2024-03-29T11:20:19.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.974+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f060801b130 con 0x7f061013ef30 2024-03-29T11:20:19.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.974+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f0608054cd0 con 0x7f061013ef30 2024-03-29T11:20:19.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.974+0000 7f05f7fff640 10 client.4244.objecter ms_dispatch 0x7f0610000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:19.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.974+0000 7f05f7fff640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:19.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.974+0000 7f05f7fff640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:19.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.974+0000 7f05f7fff640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-29T11:20:19.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.976+0000 7f060f7fe640 10 monclient: get_auth_request con 0x7f05fc06dfa0 auth_method 0 2024-03-29T11:20:19.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.978+0000 7f05f7fff640 10 client.4244.objecter ms_handle_connect 0x7f05fc06dfa0 2024-03-29T11:20:19.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.984+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0608050f70 con 0x7f061013ef30 2024-03-29T11:20:19.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.984+0000 7f05f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:19.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:19.984+0000 7f05f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:20.287 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.309+0000 7f06161a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:20.287 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.309+0000 7f06161a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:20.287 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.309+0000 7f06161a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:20.287 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.309+0000 7f06161a3640 1 -- 172.21.15.59:0/407001027 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0610138550 con 0x7f061013ef30 2024-03-29T11:20:20.290 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.313+0000 7f05f7fff640 1 -- 172.21.15.59:0/407001027 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f0610138550 con 0x7f061013ef30 2024-03-29T11:20:20.290 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.313+0000 7f05f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:20.290 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.313+0000 7f05f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:20.290 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:20:20.291 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:20.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.316+0000 7f06161a3640 10 librados: watch_flush enter 2024-03-29T11:20:20.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.316+0000 7f06161a3640 10 librados: watch_flush exit 2024-03-29T11:20:20.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.316+0000 7f06161a3640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-29T11:20:20.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.316+0000 7f06161a3640 10 client.4244.objecter successfully canceled tick 2024-03-29T11:20:20.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.316+0000 7f06161a3640 1 -- 172.21.15.59:0/407001027 >> v1:172.21.15.172:6816/2420742773 conn(0x7f05fc06dfa0 legacy=0x7f05fc070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.316+0000 7f06161a3640 10 monclient: shutdown 2024-03-29T11:20:20.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.316+0000 7f06161a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:20.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.316+0000 7f06161a3640 1 -- 172.21.15.59:0/407001027 >> v1:172.21.15.59:6790/0 conn(0x7f061013ef30 legacy=0x7f06101dbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.316+0000 7f06161a3640 1 -- 172.21.15.59:0/407001027 shutdown_connections 2024-03-29T11:20:20.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.316+0000 7f06161a3640 1 -- 172.21.15.59:0/407001027 >> 172.21.15.59:0/407001027 conn(0x7f061006b660 msgr2=0x7f061013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:20.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.317+0000 7f06161a3640 1 -- 172.21.15.59:0/407001027 shutdown_connections 2024-03-29T11:20:20.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.317+0000 7f06161a3640 1 -- 172.21.15.59:0/407001027 wait complete. 2024-03-29T11:20:20.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:20.317+0000 7f06161a3640 1 librados: shutdown 2024-03-29T11:20:20.312 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:27.118 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:27.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.303+0000 7f17c0e57640 10 monclient: get_monmap_and_config 2024-03-29T11:20:27.281 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.303+0000 7f17c0e57640 10 monclient: build_initial_monmap 2024-03-29T11:20:27.281 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.303+0000 7f17c0e57640 10 monclient: monmap: 2024-03-29T11:20:27.281 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:27.281 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:27.281 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:20:27.304847+0000 2024-03-29T11:20:27.281 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:20:27.304847+0000 2024-03-29T11:20:27.281 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:27.281 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:27.281 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:27.282 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:20:27.282 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:20:27.282 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:27.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.303+0000 7f17c0e57640 1 Processor -- start 2024-03-29T11:20:27.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.304+0000 7f17c0e57640 1 -- start start 2024-03-29T11:20:27.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.304+0000 7f17c0e57640 10 monclient: init 2024-03-29T11:20:27.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.305+0000 7f17c0e57640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:27.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.305+0000 7f17c0e57640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:27.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.305+0000 7f17c0e57640 10 monclient(hunting): picked mon.noname-b con 0x7f17bc0b4330 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:27.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.305+0000 7f17c0e57640 10 monclient(hunting): picked mon.noname-c con 0x7f17bc1470b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:27.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.305+0000 7f17c0e57640 10 monclient(hunting): picked mon.noname-a con 0x7f17bc1439b0 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:27.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.305+0000 7f17c0e57640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc06f2a0 con 0x7f17bc1439b0 2024-03-29T11:20:27.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.305+0000 7f17c0e57640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc146b90 con 0x7f17bc1470b0 2024-03-29T11:20:27.285 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.305+0000 7f17c0e57640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc0b2ca0 con 0x7f17bc0b4330 2024-03-29T11:20:27.285 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.305+0000 7f17c0e57640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:27.285 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.305+0000 7f17c0e57640 10 monclient(hunting): authenticate will time out at 877.419968s 2024-03-29T11:20:27.286 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.308+0000 7f17b9d9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f17bc1470b0 0x7f17bc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:54267/0 (socket says 172.21.15.59:54267) 2024-03-29T11:20:27.287 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.308+0000 7f17b9d9b640 1 -- 172.21.15.59:0/2578719349 learned_addr learned my addr 172.21.15.59:0/2578719349 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:27.287 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.309+0000 7f17b959a640 1 -- 172.21.15.59:0/2578719349 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701332483 0 0) 0x7f17bc146b90 con 0x7f17bc1470b0 2024-03-29T11:20:27.287 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.309+0000 7f17b959a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:27.287 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.309+0000 7f17b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:27.287 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.309+0000 7f17b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:27.287 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.309+0000 7f17b959a640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:20:27.287 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.309+0000 7f17b959a640 1 -- 172.21.15.59:0/2578719349 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a4003520 con 0x7f17bc1470b0 2024-03-29T11:20:27.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 1 -- 172.21.15.59:0/2578719349 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1100362494 0 0) 0x7f17a4003520 con 0x7f17bc1470b0 2024-03-29T11:20:27.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 1 -- 172.21.15.59:0/2578719349 >> v1:172.21.15.59:6790/0 conn(0x7f17bc0b4330 legacy=0x7f17bc0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:27.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 1 -- 172.21.15.59:0/2578719349 >> v1:172.21.15.59:6789/0 conn(0x7f17bc1439b0 legacy=0x7f17bc151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:27.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:27.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 1 monclient: found mon.noname-c 2024-03-29T11:20:27.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:27.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:20:27.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 1 -- 172.21.15.59:0/2578719349 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc06e0c0 con 0x7f17bc1470b0 2024-03-29T11:20:27.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 10 monclient: _finish_auth 0 2024-03-29T11:20:27.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:27.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:27.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 1 -- 172.21.15.59:0/2578719349 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f17b0001690 con 0x7f17bc1470b0 2024-03-29T11:20:27.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:27.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 10 monclient: dump: 2024-03-29T11:20:27.296 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:27.296 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:27.296 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.296 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.296 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.296 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:27.296 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:27.296 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:27.296 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:27.296 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:27.296 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 1 -- 172.21.15.59:0/2578719349 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17b0001860 con 0x7f17bc1470b0 2024-03-29T11:20:27.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:27.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 1 -- 172.21.15.59:0/2578719349 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f17b0001cb0 con 0x7f17bc1470b0 2024-03-29T11:20:27.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.313+0000 7f17b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.314+0000 7f17b959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:27.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.314+0000 7f17b959a640 10 monclient: dump: 2024-03-29T11:20:27.297 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:27.297 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:27.297 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.297 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.297 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.297 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:27.298 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:27.298 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:27.298 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:27.298 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:27.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.314+0000 7f17c0e57640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:20:27.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.314+0000 7f17c0e57640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:27.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.314+0000 7f17c0e57640 4 set_mon_vals no callback set 2024-03-29T11:20:27.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.315+0000 7f17c0e57640 10 monclient: shutdown 2024-03-29T11:20:27.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.315+0000 7f17c0e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:27.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.315+0000 7f17c0e57640 1 -- 172.21.15.59:0/2578719349 >> v1:172.21.15.172:6789/0 conn(0x7f17bc1470b0 legacy=0x7f17bc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.316+0000 7f17c0e57640 1 -- 172.21.15.59:0/2578719349 shutdown_connections 2024-03-29T11:20:27.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.316+0000 7f17c0e57640 1 -- 172.21.15.59:0/2578719349 >> 172.21.15.59:0/2578719349 conn(0x7f17bc066a00 msgr2=0x7f17bc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:27.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.316+0000 7f17c0e57640 1 -- 172.21.15.59:0/2578719349 shutdown_connections 2024-03-29T11:20:27.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.316+0000 7f17c0e57640 1 -- 172.21.15.59:0/2578719349 wait complete. 2024-03-29T11:20:27.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.317+0000 7f17c0e57640 10 monclient: build_initial_monmap 2024-03-29T11:20:27.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.317+0000 7f17c0e57640 10 monclient: monmap: 2024-03-29T11:20:27.303 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:27.303 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:27.303 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:20:27.304 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:20:27.304 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:27.304 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:27.304 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:27.304 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:20:27.304 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:20:27.304 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:27.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.318+0000 7f17c0e57640 1 librados: starting msgr at 2024-03-29T11:20:27.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.318+0000 7f17c0e57640 1 librados: starting objecter 2024-03-29T11:20:27.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.318+0000 7f17c0e57640 1 Processor -- start 2024-03-29T11:20:27.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 1 -- start start 2024-03-29T11:20:27.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 1 librados: setting wanted keys 2024-03-29T11:20:27.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 1 librados: calling monclient init 2024-03-29T11:20:27.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 10 monclient: init 2024-03-29T11:20:27.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:27.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:27.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 10 monclient(hunting): picked mon.noname-b con 0x7f17bc0b4330 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:27.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 10 monclient(hunting): picked mon.noname-a con 0x7f17bc1439b0 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:27.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 10 monclient(hunting): picked mon.noname-c con 0x7f17bc1470b0 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:27.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc140b80 con 0x7f17bc1439b0 2024-03-29T11:20:27.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc1dba70 con 0x7f17bc0b4330 2024-03-29T11:20:27.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc1dbc00 con 0x7f17bc1470b0 2024-03-29T11:20:27.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:27.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.319+0000 7f17c0e57640 10 monclient(hunting): authenticate will time out at 877.434598s 2024-03-29T11:20:27.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.323+0000 7f17ba59c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f17bc0b4330 0x7f17bc1db360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:54351/0 (socket says 172.21.15.59:54351) 2024-03-29T11:20:27.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.323+0000 7f17ba59c640 1 -- 172.21.15.59:0/4077355902 learned_addr learned my addr 172.21.15.59:0/4077355902 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:27.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.323+0000 7f17aaffd640 10 client.?.objecter ms_handle_connect 0x7f17bc0b4330 2024-03-29T11:20:27.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.323+0000 7f17aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:27.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.324+0000 7f17aaffd640 10 client.?.objecter ms_handle_connect 0x7f17bc1439b0 2024-03-29T11:20:27.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.324+0000 7f17aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:27.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.324+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233658527 0 0) 0x7f17bc1dba70 con 0x7f17bc0b4330 2024-03-29T11:20:27.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.324+0000 7f17aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:27.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.324+0000 7f17aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:27.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.324+0000 7f17aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:27.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.324+0000 7f17aaffd640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:20:27.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.324+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179c004eb0 con 0x7f17bc0b4330 2024-03-29T11:20:27.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.324+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159461454 0 0) 0x7f17bc140b80 con 0x7f17bc1439b0 2024-03-29T11:20:27.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.324+0000 7f17aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:27.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.325+0000 7f17aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:27.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.325+0000 7f17aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:27.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.325+0000 7f17aaffd640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:20:27.308 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.325+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17bc1dba70 con 0x7f17bc1439b0 2024-03-29T11:20:27.309 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.328+0000 7f17aaffd640 10 client.?.objecter ms_handle_connect 0x7f17bc1470b0 2024-03-29T11:20:27.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.328+0000 7f17aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:27.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.328+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267741698 0 0) 0x7f17bc1dbc00 con 0x7f17bc1470b0 2024-03-29T11:20:27.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.328+0000 7f17aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:27.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.328+0000 7f17aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:27.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.328+0000 7f17aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:27.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.328+0000 7f17aaffd640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:20:27.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.328+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17bc140b80 con 0x7f17bc1470b0 2024-03-29T11:20:27.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.329+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 261171560 0 0) 0x7f17bc1dba70 con 0x7f17bc1439b0 2024-03-29T11:20:27.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.329+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17bc1dbc00 con 0x7f17bc1439b0 2024-03-29T11:20:27.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.329+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2492150029 0 0) 0x7f17bc140b80 con 0x7f17bc1470b0 2024-03-29T11:20:27.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.329+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17bc1dba70 con 0x7f17bc1470b0 2024-03-29T11:20:27.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.329+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f17b0001b70 con 0x7f17bc1439b0 2024-03-29T11:20:27.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:27.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient(hunting): dump: 2024-03-29T11:20:27.313 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:27.313 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:27.313 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.313 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.313 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.313 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:27.313 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:27.313 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f17ac001670 con 0x7f17bc1470b0 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient(hunting): dump: 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.314 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.315 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:27.315 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:27.315 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:27.315 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:27.315 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:27.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1836650993 0 0) 0x7f179c004eb0 con 0x7f17bc0b4330 2024-03-29T11:20:27.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17bc140b80 con 0x7f17bc0b4330 2024-03-29T11:20:27.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f17b4004360 con 0x7f17bc0b4330 2024-03-29T11:20:27.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:27.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient(hunting): dump: 2024-03-29T11:20:27.316 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:27.316 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:27.316 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.316 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.316 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.316 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:27.316 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:27.316 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:27.316 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:27.316 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:27.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3725602518 0 0) 0x7f17bc1dbc00 con 0x7f17bc1439b0 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 >> v1:172.21.15.59:6790/0 conn(0x7f17bc1470b0 legacy=0x7f17bc140470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 >> v1:172.21.15.172:6789/0 conn(0x7f17bc0b4330 legacy=0x7f17bc1db360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 1 monclient: found mon.a 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc120320 con 0x7f17bc1439b0 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.330+0000 7f17aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17c0e57640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:20:27.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17c0e57640 10 monclient: _renew_subs 2024-03-29T11:20:27.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17c0e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:27.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17c0e57640 1 -- 172.21.15.59:0/4077355902 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17bc1205d0 con 0x7f17bc1439b0 2024-03-29T11:20:27.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17c0e57640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:27.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17c0e57640 10 monclient: _renew_subs 2024-03-29T11:20:27.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17c0e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:27.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17c0e57640 1 -- 172.21.15.59:0/4077355902 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17bc141080 con 0x7f17bc1439b0 2024-03-29T11:20:27.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17b0004bf0 con 0x7f17bc1439b0 2024-03-29T11:20:27.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17c0e57640 1 librados: init done 2024-03-29T11:20:27.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:27.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f17b0004f00 con 0x7f17bc1439b0 2024-03-29T11:20:27.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:27.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17aaffd640 10 monclient: dump: 2024-03-29T11:20:27.319 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:27.319 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:27.319 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.319 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:27.319 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.319 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:27.319 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:27.319 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:27.320 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:27.320 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:27.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.331+0000 7f17b8d99640 4 set_mon_vals no callback set 2024-03-29T11:20:27.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.334+0000 7f17c0e57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:27.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.334+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f17b001aed0 con 0x7f17bc1439b0 2024-03-29T11:20:27.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.334+0000 7f17c0e57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:27.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.334+0000 7f17c0e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:27.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.334+0000 7f17c0e57640 1 -- 172.21.15.59:0/4077355902 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a0005080 con 0x7f17bc1439b0 2024-03-29T11:20:27.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.335+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f17b0084b40 con 0x7f17bc1439b0 2024-03-29T11:20:27.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.335+0000 7f17aaffd640 10 client.4260.objecter ms_dispatch 0x7f17bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:27.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.335+0000 7f17aaffd640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:27.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.335+0000 7f17aaffd640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:27.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.335+0000 7f17aaffd640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T11:20:27.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.335+0000 7f17ba59c640 10 monclient: get_auth_request con 0x7f179c06e170 auth_method 0 2024-03-29T11:20:27.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.336+0000 7f17aaffd640 10 client.4260.objecter ms_handle_connect 0x7f179c06e170 2024-03-29T11:20:27.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.342+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17b0050420 con 0x7f17bc1439b0 2024-03-29T11:20:27.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.342+0000 7f17aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:27.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.342+0000 7f17aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:27.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.661+0000 7f17c0e57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:27.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.661+0000 7f17c0e57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:27.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.662+0000 7f17c0e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:27.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.662+0000 7f17c0e57640 1 -- 172.21.15.59:0/4077355902 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17a0005330 con 0x7f17bc1439b0 2024-03-29T11:20:27.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.663+0000 7f17aaffd640 1 -- 172.21.15.59:0/4077355902 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f17b0052350 con 0x7f17bc1439b0 2024-03-29T11:20:27.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.663+0000 7f17aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:27.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.663+0000 7f17aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:27.642 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:20:27.642 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:27.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.666+0000 7f17c0e57640 10 librados: watch_flush enter 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.666+0000 7f17c0e57640 10 librados: watch_flush exit 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.666+0000 7f17c0e57640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.666+0000 7f17c0e57640 10 client.4260.objecter successfully canceled tick 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.666+0000 7f17c0e57640 1 -- 172.21.15.59:0/4077355902 >> v1:172.21.15.172:6816/2420742773 conn(0x7f179c06e170 legacy=0x7f179c070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.666+0000 7f17c0e57640 10 monclient: shutdown 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.666+0000 7f17c0e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.666+0000 7f17c0e57640 1 -- 172.21.15.59:0/4077355902 >> v1:172.21.15.59:6789/0 conn(0x7f17bc1439b0 legacy=0x7f17bc13fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.667+0000 7f17c0e57640 1 -- 172.21.15.59:0/4077355902 shutdown_connections 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.667+0000 7f17c0e57640 1 -- 172.21.15.59:0/4077355902 >> 172.21.15.59:0/4077355902 conn(0x7f17bc066a00 msgr2=0x7f17bc0b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.667+0000 7f17c0e57640 1 -- 172.21.15.59:0/4077355902 shutdown_connections 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.667+0000 7f17c0e57640 1 -- 172.21.15.59:0/4077355902 wait complete. 2024-03-29T11:20:27.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:27.667+0000 7f17c0e57640 1 librados: shutdown 2024-03-29T11:20:27.661 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:34.469 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.641+0000 7f26a707b640 10 monclient: get_monmap_and_config 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.641+0000 7f26a707b640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.641+0000 7f26a707b640 10 monclient: monmap: 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:20:34.643705+0000 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:20:34.643705+0000 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:20:34.620 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:20:34.621 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:34.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.646+0000 7f26a707b640 1 Processor -- start 2024-03-29T11:20:34.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.646+0000 7f26a707b640 1 -- start start 2024-03-29T11:20:34.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.646+0000 7f26a707b640 10 monclient: init 2024-03-29T11:20:34.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.647+0000 7f26a707b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.647+0000 7f26a707b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:34.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.647+0000 7f26a707b640 10 monclient(hunting): picked mon.noname-a con 0x7f26a00a9ac0 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:34.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.647+0000 7f26a707b640 10 monclient(hunting): picked mon.noname-b con 0x7f26a013ee90 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:34.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.647+0000 7f26a707b640 10 monclient(hunting): picked mon.noname-c con 0x7f26a00af900 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:34.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.647+0000 7f26a707b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a00afd10 con 0x7f26a00a9ac0 2024-03-29T11:20:34.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.647+0000 7f26a707b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a00af6f0 con 0x7f26a00af900 2024-03-29T11:20:34.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.647+0000 7f26a707b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a00aac80 con 0x7f26a013ee90 2024-03-29T11:20:34.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.647+0000 7f26a707b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.628 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.647+0000 7f26a707b640 10 monclient(hunting): authenticate will time out at 884.762491s 2024-03-29T11:20:34.630 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.652+0000 7f26a5618640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f26a00af900 0x7f26a0149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:40623/0 (socket says 172.21.15.59:40623) 2024-03-29T11:20:34.630 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.653+0000 7f26a5618640 1 -- 172.21.15.59:0/945827025 learned_addr learned my addr 172.21.15.59:0/945827025 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:34.631 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.653+0000 7f269f7fe640 1 -- 172.21.15.59:0/945827025 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446339622 0 0) 0x7f26a00afd10 con 0x7f26a00a9ac0 2024-03-29T11:20:34.631 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.653+0000 7f269f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.631 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.653+0000 7f269f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.631 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.653+0000 7f269f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.631 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.653+0000 7f269f7fe640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:20:34.632 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.653+0000 7f269f7fe640 1 -- 172.21.15.59:0/945827025 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2688003520 con 0x7f26a00a9ac0 2024-03-29T11:20:34.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 1 -- 172.21.15.59:0/945827025 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2757057699 0 0) 0x7f2688003520 con 0x7f26a00a9ac0 2024-03-29T11:20:34.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 1 -- 172.21.15.59:0/945827025 >> v1:172.21.15.59:6790/0 conn(0x7f26a013ee90 legacy=0x7f26a013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 1 -- 172.21.15.59:0/945827025 >> v1:172.21.15.172:6789/0 conn(0x7f26a00af900 legacy=0x7f26a0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:20:34.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:20:34.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 1 -- 172.21.15.59:0/945827025 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a00aaf60 con 0x7f26a00a9ac0 2024-03-29T11:20:34.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 1 -- 172.21.15.59:0/945827025 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f26980019f0 con 0x7f26a00a9ac0 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.655+0000 7f269f7fe640 10 monclient: dump: 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:34.638 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.658+0000 7f26a707b640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.658+0000 7f26a707b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.658+0000 7f269f7fe640 1 -- 172.21.15.59:0/945827025 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2698001b50 con 0x7f26a00a9ac0 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.658+0000 7f269f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.658+0000 7f269f7fe640 1 -- 172.21.15.59:0/945827025 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2698001fa0 con 0x7f26a00a9ac0 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.658+0000 7f269f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.658+0000 7f269f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.658+0000 7f269f7fe640 10 monclient: dump: 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.639 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:34.640 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:34.640 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:34.640 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:34.640 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:34.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.658+0000 7f26a707b640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:34.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.658+0000 7f26a707b640 4 set_mon_vals no callback set 2024-03-29T11:20:34.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.659+0000 7f26a707b640 10 monclient: shutdown 2024-03-29T11:20:34.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.659+0000 7f26a707b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.659+0000 7f26a707b640 1 -- 172.21.15.59:0/945827025 >> v1:172.21.15.59:6789/0 conn(0x7f26a00a9ac0 legacy=0x7f26a00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.660+0000 7f26a707b640 1 -- 172.21.15.59:0/945827025 shutdown_connections 2024-03-29T11:20:34.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.660+0000 7f26a707b640 1 -- 172.21.15.59:0/945827025 >> 172.21.15.59:0/945827025 conn(0x7f26a00b35e0 msgr2=0x7f26a00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.663+0000 7f26a707b640 1 -- 172.21.15.59:0/945827025 shutdown_connections 2024-03-29T11:20:34.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.663+0000 7f26a707b640 1 -- 172.21.15.59:0/945827025 wait complete. 2024-03-29T11:20:34.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.664+0000 7f26a707b640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.664+0000 7f26a707b640 10 monclient: monmap: 2024-03-29T11:20:34.642 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:34.642 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:34.642 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:20:34.642 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:20:34.642 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.664+0000 7f26a707b640 1 librados: starting msgr at 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.664+0000 7f26a707b640 1 librados: starting objecter 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.664+0000 7f26a707b640 1 Processor -- start 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.665+0000 7f26a707b640 1 -- start start 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.665+0000 7f26a707b640 1 librados: setting wanted keys 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.665+0000 7f26a707b640 1 librados: calling monclient init 2024-03-29T11:20:34.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.665+0000 7f26a707b640 10 monclient: init 2024-03-29T11:20:34.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.666+0000 7f26a707b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.666+0000 7f26a707b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:34.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.666+0000 7f26a707b640 10 monclient(hunting): picked mon.noname-b con 0x7f26a00a9ac0 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:34.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.666+0000 7f26a707b640 10 monclient(hunting): picked mon.noname-a con 0x7f26a00af900 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:34.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.666+0000 7f26a707b640 10 monclient(hunting): picked mon.noname-c con 0x7f26a013ee90 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:34.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.666+0000 7f26a707b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a01e4000 con 0x7f26a00af900 2024-03-29T11:20:34.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.666+0000 7f26a707b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a01c67b0 con 0x7f26a00a9ac0 2024-03-29T11:20:34.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.666+0000 7f26a707b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a00725c0 con 0x7f26a013ee90 2024-03-29T11:20:34.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.666+0000 7f26a707b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.666+0000 7f26a707b640 10 monclient(hunting): authenticate will time out at 884.780982s 2024-03-29T11:20:34.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.672+0000 7f26a4e17640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f26a00a9ac0 0x7f26a01e1220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:40749/0 (socket says 172.21.15.59:40749) 2024-03-29T11:20:34.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.672+0000 7f26a5618640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f26a013ee90 0x7f26a01ef7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:42741/0 (socket says 172.21.15.59:42741) 2024-03-29T11:20:34.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.672+0000 7f26a5618640 1 -- 172.21.15.59:0/355294409 learned_addr learned my addr 172.21.15.59:0/355294409 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:34.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 10 client.?.objecter ms_handle_connect 0x7f26a00af900 2024-03-29T11:20:34.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 10 client.?.objecter ms_handle_connect 0x7f26a00a9ac0 2024-03-29T11:20:34.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311438154 0 0) 0x7f26a01e4000 con 0x7f26a00af900 2024-03-29T11:20:34.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:20:34.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f268c004eb0 con 0x7f26a00af900 2024-03-29T11:20:34.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 10 client.?.objecter ms_handle_connect 0x7f26a013ee90 2024-03-29T11:20:34.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.673+0000 7f269d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152848165 0 0) 0x7f26a00725c0 con 0x7f26a013ee90 2024-03-29T11:20:34.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:20:34.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a01e4000 con 0x7f26a013ee90 2024-03-29T11:20:34.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4275282472 0 0) 0x7f268c004eb0 con 0x7f26a00af900 2024-03-29T11:20:34.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26a00725c0 con 0x7f26a00af900 2024-03-29T11:20:34.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2694001850 con 0x7f26a00af900 2024-03-29T11:20:34.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 10 monclient(hunting): dump: 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:34.654 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2946030959 0 0) 0x7f26a01e4000 con 0x7f26a013ee90 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f268c004eb0 con 0x7f26a013ee90 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2690001810 con 0x7f26a013ee90 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 10 monclient(hunting): dump: 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:34.655 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:34.656 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:34.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3666622622 0 0) 0x7f268c004eb0 con 0x7f26a013ee90 2024-03-29T11:20:34.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 >> v1:172.21.15.172:6789/0 conn(0x7f26a00a9ac0 legacy=0x7f26a01e1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.674+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 >> v1:172.21.15.59:6789/0 conn(0x7f26a00af900 legacy=0x7f26a01e44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f269d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f269d7fa640 1 monclient: found mon.c 2024-03-29T11:20:34.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f269d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f269d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:34.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a006fc60 con 0x7f26a013ee90 2024-03-29T11:20:34.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f269d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f269d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f269d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f26a707b640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:20:34.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f26a707b640 10 monclient: _renew_subs 2024-03-29T11:20:34.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f26a707b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:34.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f26a707b640 1 -- 172.21.15.59:0/355294409 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a01c4c70 con 0x7f26a013ee90 2024-03-29T11:20:34.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f26a707b640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:34.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f26a707b640 10 monclient: _renew_subs 2024-03-29T11:20:34.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f26a707b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:34.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f26a707b640 1 -- 172.21.15.59:0/355294409 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a01e1b10 con 0x7f26a013ee90 2024-03-29T11:20:34.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f26a707b640 1 librados: init done 2024-03-29T11:20:34.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2690001ce0 con 0x7f26a013ee90 2024-03-29T11:20:34.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f269d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.661 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.675+0000 7f269effd640 4 set_mon_vals no callback set 2024-03-29T11:20:34.661 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.676+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2690004ab0 con 0x7f26a013ee90 2024-03-29T11:20:34.661 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.676+0000 7f269d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.661 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.676+0000 7f269d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:34.661 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.676+0000 7f269d7fa640 10 monclient: dump: 2024-03-29T11:20:34.661 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:34.661 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:34.661 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:34.661 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:34.661 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.679+0000 7f26a707b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.679+0000 7f26a707b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.679+0000 7f26a707b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.679+0000 7f26a707b640 1 -- 172.21.15.59:0/355294409 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2680005080 con 0x7f26a013ee90 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.681+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f2690004cd0 con 0x7f26a013ee90 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.681+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f2690084ad0 con 0x7f26a013ee90 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.682+0000 7f269d7fa640 10 client.4253.objecter ms_dispatch 0x7f26a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:34.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.682+0000 7f269d7fa640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:34.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.682+0000 7f26a4e17640 10 monclient: get_auth_request con 0x7f268c06de50 auth_method 0 2024-03-29T11:20:34.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.682+0000 7f269d7fa640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:34.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.682+0000 7f269d7fa640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T11:20:34.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.682+0000 7f269d7fa640 10 client.4253.objecter ms_handle_connect 0x7f268c06de50 2024-03-29T11:20:34.667 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.689+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2690086e60 con 0x7f26a013ee90 2024-03-29T11:20:34.667 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.689+0000 7f269d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.667 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:34.689+0000 7f269d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:34.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.019+0000 7f26a707b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:34.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.019+0000 7f26a707b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:34.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.019+0000 7f26a707b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:34.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.019+0000 7f26a707b640 1 -- 172.21.15.59:0/355294409 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2680005330 con 0x7f26a013ee90 2024-03-29T11:20:34.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.020+0000 7f269d7fa640 1 -- 172.21.15.59:0/355294409 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f26900503b0 con 0x7f26a013ee90 2024-03-29T11:20:34.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.020+0000 7f269d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:34.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.020+0000 7f269d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:34.998 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:20:34.998 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:35.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.023+0000 7f26a707b640 10 librados: watch_flush enter 2024-03-29T11:20:35.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.024+0000 7f26a707b640 10 librados: watch_flush exit 2024-03-29T11:20:35.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.024+0000 7f26a707b640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T11:20:35.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.024+0000 7f26a707b640 10 client.4253.objecter successfully canceled tick 2024-03-29T11:20:35.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.024+0000 7f26a707b640 1 -- 172.21.15.59:0/355294409 >> v1:172.21.15.172:6816/2420742773 conn(0x7f268c06de50 legacy=0x7f268c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.024+0000 7f26a707b640 10 monclient: shutdown 2024-03-29T11:20:35.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.024+0000 7f26a707b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.002 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.024+0000 7f26a707b640 1 -- 172.21.15.59:0/355294409 >> v1:172.21.15.59:6790/0 conn(0x7f26a013ee90 legacy=0x7f26a01ef7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.002 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.025+0000 7f26a707b640 1 -- 172.21.15.59:0/355294409 shutdown_connections 2024-03-29T11:20:35.002 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.025+0000 7f26a707b640 1 -- 172.21.15.59:0/355294409 >> 172.21.15.59:0/355294409 conn(0x7f26a00b35e0 msgr2=0x7f26a00ad300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.004 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.026+0000 7f26a707b640 1 -- 172.21.15.59:0/355294409 shutdown_connections 2024-03-29T11:20:35.004 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.026+0000 7f26a707b640 1 -- 172.21.15.59:0/355294409 wait complete. 2024-03-29T11:20:35.004 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:35.027+0000 7f26a707b640 1 librados: shutdown 2024-03-29T11:20:35.023 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:41.830 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:41.978 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.001+0000 7f9614930640 10 monclient: get_monmap_and_config 2024-03-29T11:20:41.979 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.001+0000 7f9614930640 10 monclient: build_initial_monmap 2024-03-29T11:20:41.979 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.001+0000 7f9614930640 10 monclient: monmap: 2024-03-29T11:20:41.979 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:41.979 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:41.979 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:20:42.002380+0000 2024-03-29T11:20:41.979 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:20:42.002380+0000 2024-03-29T11:20:41.979 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:41.980 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:41.980 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:41.980 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:20:41.980 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:20:41.980 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:41.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.002+0000 7f9614930640 1 Processor -- start 2024-03-29T11:20:41.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.002+0000 7f9614930640 1 -- start start 2024-03-29T11:20:41.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.002+0000 7f9614930640 10 monclient: init 2024-03-29T11:20:41.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.003+0000 7f9614930640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:41.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.003+0000 7f9614930640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:41.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.004+0000 7f9614930640 10 monclient(hunting): picked mon.noname-c con 0x7f96100b0770 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:41.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.005+0000 7f9614930640 10 monclient(hunting): picked mon.noname-a con 0x7f96100ad850 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:41.983 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.005+0000 7f9614930640 10 monclient(hunting): picked mon.noname-b con 0x7f961013c1b0 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:41.983 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.005+0000 7f9614930640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96100ac8e0 con 0x7f96100ad850 2024-03-29T11:20:41.983 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.005+0000 7f9614930640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96100ad330 con 0x7f96100b0770 2024-03-29T11:20:41.983 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.005+0000 7f9614930640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96100acfe0 con 0x7f961013c1b0 2024-03-29T11:20:41.983 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.005+0000 7f9614930640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:41.983 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.005+0000 7f9614930640 10 monclient(hunting): authenticate will time out at 892.119414s 2024-03-29T11:20:41.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.008+0000 7f960dd9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f96100ad850 0x7f96100aaae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37101/0 (socket says 172.21.15.59:37101) 2024-03-29T11:20:41.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.008+0000 7f960dd9b640 1 -- 172.21.15.59:0/292696149 learned_addr learned my addr 172.21.15.59:0/292696149 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:41.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 1 -- 172.21.15.59:0/292696149 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351156464 0 0) 0x7f96100ac8e0 con 0x7f96100ad850 2024-03-29T11:20:41.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:41.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:41.991 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:41.991 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:20:41.991 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 1 -- 172.21.15.59:0/292696149 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f8005730 con 0x7f96100ad850 2024-03-29T11:20:41.991 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 1 -- 172.21.15.59:0/292696149 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 148522897 0 0) 0x7f95f8005730 con 0x7f96100ad850 2024-03-29T11:20:41.991 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 1 -- 172.21.15.59:0/292696149 >> v1:172.21.15.59:6790/0 conn(0x7f961013c1b0 legacy=0x7f961014a760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:41.992 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 1 -- 172.21.15.59:0/292696149 >> v1:172.21.15.172:6789/0 conn(0x7f96100b0770 legacy=0x7f9610072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:41.992 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:41.992 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 1 monclient: found mon.noname-a 2024-03-29T11:20:41.992 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:41.992 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:20:41.992 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 1 -- 172.21.15.59:0/292696149 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96100ab1f0 con 0x7f96100ad850 2024-03-29T11:20:41.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient: _finish_auth 0 2024-03-29T11:20:41.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:41.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:41.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f9614930640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:20:41.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f9614930640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:41.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 1 -- 172.21.15.59:0/292696149 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9608001690 con 0x7f96100ad850 2024-03-29T11:20:41.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:41.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:41.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient: dump: 2024-03-29T11:20:41.994 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:41.994 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:41.995 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:41.995 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:41.995 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:41.995 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:41.995 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:41.995 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:41.995 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:41.996 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:41.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f9614930640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:41.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 1 -- 172.21.15.59:0/292696149 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9608001920 con 0x7f96100ad850 2024-03-29T11:20:41.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.009+0000 7f960d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:41.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.010+0000 7f9614930640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:41.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.010+0000 7f9614930640 4 set_mon_vals no callback set 2024-03-29T11:20:41.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.010+0000 7f960d59a640 1 -- 172.21.15.59:0/292696149 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9608001cf0 con 0x7f96100ad850 2024-03-29T11:20:41.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.010+0000 7f960d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:41.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.010+0000 7f960d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:41.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.010+0000 7f960d59a640 10 monclient: dump: 2024-03-29T11:20:41.997 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:41.997 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:41.998 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:41.998 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:41.998 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:41.998 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:41.998 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:41.998 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:41.999 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:41.999 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:42.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.010+0000 7f9614930640 10 monclient: shutdown 2024-03-29T11:20:42.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.011+0000 7f9614930640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:42.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.011+0000 7f9614930640 1 -- 172.21.15.59:0/292696149 >> v1:172.21.15.59:6789/0 conn(0x7f96100ad850 legacy=0x7f96100aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.014+0000 7f9614930640 1 -- 172.21.15.59:0/292696149 shutdown_connections 2024-03-29T11:20:42.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.014+0000 7f9614930640 1 -- 172.21.15.59:0/292696149 >> 172.21.15.59:0/292696149 conn(0x7f961006b660 msgr2=0x7f961006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:42.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.014+0000 7f9614930640 1 -- 172.21.15.59:0/292696149 shutdown_connections 2024-03-29T11:20:42.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.015+0000 7f9614930640 1 -- 172.21.15.59:0/292696149 wait complete. 2024-03-29T11:20:42.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.016+0000 7f9614930640 10 monclient: build_initial_monmap 2024-03-29T11:20:42.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.016+0000 7f9614930640 10 monclient: monmap: 2024-03-29T11:20:42.010 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.017+0000 7f9614930640 1 librados: starting msgr at 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.017+0000 7f9614930640 1 librados: starting objecter 2024-03-29T11:20:42.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.017+0000 7f9614930640 1 Processor -- start 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.017+0000 7f9614930640 1 -- start start 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.017+0000 7f9614930640 1 librados: setting wanted keys 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.017+0000 7f9614930640 1 librados: calling monclient init 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.017+0000 7f9614930640 10 monclient: init 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.018+0000 7f9614930640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.018+0000 7f9614930640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.018+0000 7f9614930640 10 monclient(hunting): picked mon.noname-c con 0x7f96100ad850 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.018+0000 7f9614930640 10 monclient(hunting): picked mon.noname-b con 0x7f96100b0770 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.018+0000 7f9614930640 10 monclient(hunting): picked mon.noname-a con 0x7f961013c1b0 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.018+0000 7f9614930640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961006eea0 con 0x7f961013c1b0 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.018+0000 7f9614930640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96101d2bb0 con 0x7f96100b0770 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.018+0000 7f9614930640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9610070060 con 0x7f96100ad850 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.018+0000 7f9614930640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:42.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.018+0000 7f9614930640 10 monclient(hunting): authenticate will time out at 892.131971s 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.021+0000 7f960ed9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f961013c1b0 0x7f9610070d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37221/0 (socket says 172.21.15.59:37221) 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.021+0000 7f960ed9d640 1 -- 172.21.15.59:0/364948864 learned_addr learned my addr 172.21.15.59:0/364948864 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.021+0000 7f95feffd640 10 client.?.objecter ms_handle_connect 0x7f961013c1b0 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.021+0000 7f95feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.022+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171247381 0 0) 0x7f961006eea0 con 0x7f961013c1b0 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.022+0000 7f95feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.022+0000 7f95feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.022+0000 7f95feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.022+0000 7f95feffd640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.022+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f0004eb0 con 0x7f961013c1b0 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.023+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 156976502 0 0) 0x7f95f0004eb0 con 0x7f961013c1b0 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.023+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f961006eea0 con 0x7f961013c1b0 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.023+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9600001ad0 con 0x7f961013c1b0 2024-03-29T11:20:42.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.023+0000 7f95feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.023+0000 7f95feffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.023+0000 7f95feffd640 10 monclient(hunting): dump: 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:42.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.023+0000 7f95feffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.023+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2383551011 0 0) 0x7f961006eea0 con 0x7f961013c1b0 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.023+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 >> v1:172.21.15.59:6790/0 conn(0x7f96100ad850 legacy=0x7f961006e790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 >> v1:172.21.15.172:6789/0 conn(0x7f96100b0770 legacy=0x7f96101d3090 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 1 monclient: found mon.a 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96101d37a0 con 0x7f961013c1b0 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f9614930640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f9614930640 10 monclient: _renew_subs 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f9614930640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:42.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f9614930640 1 -- 172.21.15.59:0/364948864 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961006e040 con 0x7f961013c1b0 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96000020a0 con 0x7f961013c1b0 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9600004d80 con 0x7f961013c1b0 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f9614930640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f95feffd640 10 monclient: dump: 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:42.016 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f9614930640 10 monclient: _renew_subs 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f9614930640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f9614930640 1 -- 172.21.15.59:0/364948864 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9610070660 con 0x7f961013c1b0 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f9614930640 1 librados: init done 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.024+0000 7f960cd99640 4 set_mon_vals no callback set 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.026+0000 7f9614930640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.026+0000 7f9614930640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.026+0000 7f9614930640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.026+0000 7f9614930640 1 -- 172.21.15.59:0/364948864 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f4005080 con 0x7f961013c1b0 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.027+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f9600004ee0 con 0x7f961013c1b0 2024-03-29T11:20:42.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.027+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f9600084d90 con 0x7f961013c1b0 2024-03-29T11:20:42.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.027+0000 7f95feffd640 10 client.4272.objecter ms_dispatch 0x7f9610000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:42.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.027+0000 7f95feffd640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:42.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.027+0000 7f95feffd640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:42.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.027+0000 7f95feffd640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-29T11:20:42.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.028+0000 7f960e59c640 10 monclient: get_auth_request con 0x7f95f006dc70 auth_method 0 2024-03-29T11:20:42.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.030+0000 7f95feffd640 10 client.4272.objecter ms_handle_connect 0x7f95f006dc70 2024-03-29T11:20:42.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.037+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96000504a0 con 0x7f961013c1b0 2024-03-29T11:20:42.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.037+0000 7f95feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.037+0000 7f95feffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:42.350 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.374+0000 7f9614930640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:42.350 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.374+0000 7f9614930640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:42.350 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.374+0000 7f9614930640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:20:42.351 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.374+0000 7f9614930640 1 -- 172.21.15.59:0/364948864 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95f4005330 con 0x7f961013c1b0 2024-03-29T11:20:42.351 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.374+0000 7f95feffd640 1 -- 172.21.15.59:0/364948864 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f96000525a0 con 0x7f961013c1b0 2024-03-29T11:20:42.351 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.374+0000 7f95feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:42.351 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.374+0000 7f95feffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:42.352 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:20:42.352 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:42.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.379+0000 7f9614930640 10 librados: watch_flush enter 2024-03-29T11:20:42.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 10 librados: watch_flush exit 2024-03-29T11:20:42.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-29T11:20:42.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 10 client.4272.objecter successfully canceled tick 2024-03-29T11:20:42.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 1 -- 172.21.15.59:0/364948864 >> v1:172.21.15.172:6816/2420742773 conn(0x7f95f006dc70 legacy=0x7f95f0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 10 monclient: shutdown 2024-03-29T11:20:42.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:42.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 1 -- 172.21.15.59:0/364948864 >> v1:172.21.15.59:6789/0 conn(0x7f961013c1b0 legacy=0x7f9610070d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 1 -- 172.21.15.59:0/364948864 shutdown_connections 2024-03-29T11:20:42.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 1 -- 172.21.15.59:0/364948864 >> 172.21.15.59:0/364948864 conn(0x7f961006b660 msgr2=0x7f9610135d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:42.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 1 -- 172.21.15.59:0/364948864 shutdown_connections 2024-03-29T11:20:42.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 1 -- 172.21.15.59:0/364948864 wait complete. 2024-03-29T11:20:42.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:42.381+0000 7f9614930640 1 librados: shutdown 2024-03-29T11:20:42.436 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:49.243 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:49.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.424+0000 7fac3f796640 10 monclient: get_monmap_and_config 2024-03-29T11:20:49.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.424+0000 7fac3f796640 10 monclient: build_initial_monmap 2024-03-29T11:20:49.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.424+0000 7fac3f796640 10 monclient: monmap: 2024-03-29T11:20:49.401 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:49.402 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:49.402 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:20:49.425323+0000 2024-03-29T11:20:49.402 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:20:49.425323+0000 2024-03-29T11:20:49.402 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:49.402 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:49.402 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:49.402 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:20:49.402 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:20:49.402 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:49.402 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.425+0000 7fac3f796640 1 Processor -- start 2024-03-29T11:20:49.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.425+0000 7fac3f796640 1 -- start start 2024-03-29T11:20:49.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.425+0000 7fac3f796640 10 monclient: init 2024-03-29T11:20:49.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.426+0000 7fac3f796640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:49.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.426+0000 7fac3f796640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:49.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.426+0000 7fac3f796640 10 monclient(hunting): picked mon.noname-b con 0x7fac380b46b0 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:49.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.426+0000 7fac3f796640 10 monclient(hunting): picked mon.noname-a con 0x7fac380aeae0 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:49.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.426+0000 7fac3f796640 10 monclient(hunting): picked mon.noname-c con 0x7fac3813fa10 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:49.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.426+0000 7fac3f796640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380b06a0 con 0x7fac380aeae0 2024-03-29T11:20:49.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.426+0000 7fac3f796640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380b4230 con 0x7fac3813fa10 2024-03-29T11:20:49.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac3f796640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380b0910 con 0x7fac380b46b0 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac3f796640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac3f796640 10 monclient(hunting): authenticate will time out at 899.540802s 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac3d532640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fac380b46b0 0x7fac380b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:50666/0 (socket says 172.21.15.59:50666) 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac3d532640 1 -- 172.21.15.59:0/1952832250 learned_addr learned my addr 172.21.15.59:0/1952832250 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081282006 0 0) 0x7fac380b4230 con 0x7fac3813fa10 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac24003520 con 0x7fac3813fa10 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135949198 0 0) 0x7fac380b0910 con 0x7fac380b46b0 2024-03-29T11:20:49.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac380b4230 con 0x7fac380b46b0 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893123713 0 0) 0x7fac380b06a0 con 0x7fac380aeae0 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac240039d0 con 0x7fac380aeae0 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.427+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1688742592 0 0) 0x7fac24003520 con 0x7fac3813fa10 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 >> v1:172.21.15.59:6790/0 conn(0x7fac380b46b0 legacy=0x7fac380b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 >> v1:172.21.15.59:6789/0 conn(0x7fac380aeae0 legacy=0x7fac3813f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 1 monclient: found mon.noname-c 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac380af6d0 con 0x7fac3813fa10 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac3f796640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fac30001a10 con 0x7fac3813fa10 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac3f796640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:49.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 10 monclient: dump: 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac3f796640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac30001b70 con 0x7fac3813fa10 2024-03-29T11:20:49.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 1 -- 172.21.15.59:0/1952832250 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fac30001f80 con 0x7fac3813fa10 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac3f796640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac3f796640 4 set_mon_vals no callback set 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.428+0000 7fac2ffff640 10 monclient: dump: 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:49.410 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:49.411 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:49.411 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:49.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.429+0000 7fac3f796640 10 monclient: shutdown 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.429+0000 7fac3f796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.429+0000 7fac3f796640 1 -- 172.21.15.59:0/1952832250 >> v1:172.21.15.172:6789/0 conn(0x7fac3813fa10 legacy=0x7fac3814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.429+0000 7fac3f796640 1 -- 172.21.15.59:0/1952832250 shutdown_connections 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.429+0000 7fac3f796640 1 -- 172.21.15.59:0/1952832250 >> 172.21.15.59:0/1952832250 conn(0x7fac38066a00 msgr2=0x7fac38066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.429+0000 7fac3f796640 1 -- 172.21.15.59:0/1952832250 shutdown_connections 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.429+0000 7fac3f796640 1 -- 172.21.15.59:0/1952832250 wait complete. 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.430+0000 7fac3f796640 10 monclient: build_initial_monmap 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.430+0000 7fac3f796640 10 monclient: monmap: 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:20:49.416 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:49.417 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:49.417 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:49.417 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:20:49.417 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:20:49.417 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:49.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.431+0000 7fac3f796640 1 librados: starting msgr at 2024-03-29T11:20:49.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.431+0000 7fac3f796640 1 librados: starting objecter 2024-03-29T11:20:49.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.431+0000 7fac3f796640 1 Processor -- start 2024-03-29T11:20:49.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.431+0000 7fac3f796640 1 -- start start 2024-03-29T11:20:49.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.431+0000 7fac3f796640 1 librados: setting wanted keys 2024-03-29T11:20:49.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.431+0000 7fac3f796640 1 librados: calling monclient init 2024-03-29T11:20:49.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.431+0000 7fac3f796640 10 monclient: init 2024-03-29T11:20:49.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.432+0000 7fac3f796640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:49.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.432+0000 7fac3f796640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:49.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.432+0000 7fac3f796640 10 monclient(hunting): picked mon.noname-c con 0x7fac380aeae0 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:49.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac3d532640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fac380aeae0 0x7fac38140d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:50858/0 (socket says 172.21.15.59:50858) 2024-03-29T11:20:49.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac3f796640 10 monclient(hunting): picked mon.noname-b con 0x7fac380b46b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:49.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac3cd31640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fac380b46b0 0x7fac381d75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:54888/0 (socket says 172.21.15.59:54888) 2024-03-29T11:20:49.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac3f796640 10 monclient(hunting): picked mon.noname-a con 0x7fac3813fa10 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:49.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac3d532640 1 -- 172.21.15.59:0/3227988486 learned_addr learned my addr 172.21.15.59:0/3227988486 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3813fee0 con 0x7fac3813fa10 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac38141a20 con 0x7fac380b46b0 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac38141bb0 con 0x7fac380aeae0 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac3f796640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac3f796640 10 monclient(hunting): authenticate will time out at 899.547120s 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac2dffb640 10 client.?.objecter ms_handle_connect 0x7fac380b46b0 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac2dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac2dffb640 10 client.?.objecter ms_handle_connect 0x7fac380aeae0 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.433+0000 7fac2dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 10 client.?.objecter ms_handle_connect 0x7fac3813fa10 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447546215 0 0) 0x7fac38141a20 con 0x7fac380b46b0 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:20:49.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac1c004eb0 con 0x7fac380b46b0 2024-03-29T11:20:49.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108882423 0 0) 0x7fac38141bb0 con 0x7fac380aeae0 2024-03-29T11:20:49.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:20:49.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.434+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac38141a20 con 0x7fac380aeae0 2024-03-29T11:20:49.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108223591 0 0) 0x7fac3813fee0 con 0x7fac3813fa10 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac38141bb0 con 0x7fac3813fa10 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1818847861 0 0) 0x7fac38141a20 con 0x7fac380aeae0 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac3813fee0 con 0x7fac380aeae0 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1020117300 0 0) 0x7fac1c004eb0 con 0x7fac380b46b0 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac38141a20 con 0x7fac380b46b0 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fac28001810 con 0x7fac380aeae0 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 10 monclient(hunting): dump: 2024-03-29T11:20:49.421 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:49.422 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:49.422 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:49.422 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:49.422 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.422 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:49.422 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:49.422 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:49.422 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:49.422 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.435+0000 7fac2dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.436+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fac34001850 con 0x7fac380b46b0 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.436+0000 7fac2dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.436+0000 7fac2dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.436+0000 7fac2dffb640 10 monclient(hunting): dump: 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:49.423 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:49.424 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:49.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.436+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3326919280 0 0) 0x7fac38141a20 con 0x7fac380b46b0 2024-03-29T11:20:49.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.436+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 >> v1:172.21.15.59:6790/0 conn(0x7fac380aeae0 legacy=0x7fac38140d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.436+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 >> v1:172.21.15.59:6789/0 conn(0x7fac3813fa10 legacy=0x7fac381e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.436+0000 7fac2dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:49.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.436+0000 7fac2dffb640 1 monclient: found mon.b 2024-03-29T11:20:49.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.436+0000 7fac2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:49.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.436+0000 7fac2dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:20:49.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38141d40 con 0x7fac380b46b0 2024-03-29T11:20:49.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac2dffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac2dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac3f796640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac3f796640 10 monclient: _renew_subs 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac3f796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac381d7cf0 con 0x7fac380b46b0 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac3f796640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac3f796640 10 monclient: _renew_subs 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac3f796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac340035f0 con 0x7fac380b46b0 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.437+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac381d81d0 con 0x7fac380b46b0 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.438+0000 7fac3f796640 1 librados: init done 2024-03-29T11:20:49.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.438+0000 7fac2dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:49.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.438+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fac340019b0 con 0x7fac380b46b0 2024-03-29T11:20:49.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.438+0000 7fac2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.438+0000 7fac2dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:49.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.438+0000 7fac2dffb640 10 monclient: dump: 2024-03-29T11:20:49.426 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:49.426 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:49.426 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:49.426 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:49.426 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.426 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:49.427 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:49.427 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:49.427 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:49.427 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:49.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.438+0000 7fac2f7fe640 4 set_mon_vals no callback set 2024-03-29T11:20:49.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.440+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fac340039c0 con 0x7fac380b46b0 2024-03-29T11:20:49.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.440+0000 7fac3f796640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.440+0000 7fac3f796640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.440+0000 7fac3f796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.440+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac18005080 con 0x7fac380b46b0 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.441+0000 7fac3d532640 10 monclient: get_auth_request con 0x7fac1c06e030 auth_method 0 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.441+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fac34084740 con 0x7fac380b46b0 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.441+0000 7fac2dffb640 10 client.4270.objecter ms_dispatch 0x7fac38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.441+0000 7fac2dffb640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.441+0000 7fac2dffb640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.441+0000 7fac2dffb640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.441+0000 7fac2dffb640 10 client.4270.objecter ms_handle_connect 0x7fac1c06e030 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.448+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac34050020 con 0x7fac380b46b0 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.448+0000 7fac2dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:49.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.448+0000 7fac2dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:49.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.886+0000 7fac3f796640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:49.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.886+0000 7fac3f796640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:49.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.886+0000 7fac3f796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:20:49.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.886+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac18005330 con 0x7fac380b46b0 2024-03-29T11:20:49.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.887+0000 7fac2dffb640 1 -- 172.21.15.59:0/3227988486 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fac34051f50 con 0x7fac380b46b0 2024-03-29T11:20:49.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.887+0000 7fac2dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:49.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.887+0000 7fac2dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:49.864 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:20:49.864 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:49.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 10 librados: watch_flush enter 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 10 librados: watch_flush exit 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 10 client.4270.objecter successfully canceled tick 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 >> v1:172.21.15.172:6816/2420742773 conn(0x7fac1c06e030 legacy=0x7fac1c0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 10 monclient: shutdown 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 >> v1:172.21.15.172:6789/0 conn(0x7fac380b46b0 legacy=0x7fac381d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 shutdown_connections 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 >> 172.21.15.59:0/3227988486 conn(0x7fac38066a00 msgr2=0x7fac380b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 shutdown_connections 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 1 -- 172.21.15.59:0/3227988486 wait complete. 2024-03-29T11:20:49.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:49.891+0000 7fac3f796640 1 librados: shutdown 2024-03-29T11:20:49.881 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:56.688 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:56.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.875+0000 7f1492a7e640 10 monclient: get_monmap_and_config 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.875+0000 7f1492a7e640 10 monclient: build_initial_monmap 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.875+0000 7f1492a7e640 10 monclient: monmap: 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:20:56.876179+0000 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:20:56.876179+0000 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:20:56.852 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:56.853 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.876+0000 7f1492a7e640 1 Processor -- start 2024-03-29T11:20:56.854 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.876+0000 7f1492a7e640 1 -- start start 2024-03-29T11:20:56.854 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.876+0000 7f1492a7e640 10 monclient: init 2024-03-29T11:20:56.854 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.877+0000 7f1492a7e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:56.854 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.877+0000 7f1492a7e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:56.854 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.877+0000 7f1492a7e640 10 monclient(hunting): picked mon.noname-b con 0x7f1484037210 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:56.854 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.877+0000 7f1492a7e640 10 monclient(hunting): picked mon.noname-c con 0x7f14840340d0 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:56.854 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.877+0000 7f1492a7e640 10 monclient(hunting): picked mon.noname-a con 0x7f14840c9600 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:56.854 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.877+0000 7f1492a7e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148402c1c0 con 0x7f14840c9600 2024-03-29T11:20:56.854 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.877+0000 7f1492a7e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148402c430 con 0x7f14840340d0 2024-03-29T11:20:56.854 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.877+0000 7f1492a7e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148402b080 con 0x7f1484037210 2024-03-29T11:20:56.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.877+0000 7f1492a7e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:56.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.877+0000 7f1492a7e640 10 monclient(hunting): authenticate will time out at 906.991528s 2024-03-29T11:20:56.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.878+0000 7f149227d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f14840c9600 0x7f14840cc470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54080/0 (socket says 172.21.15.59:54080) 2024-03-29T11:20:56.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.878+0000 7f149227d640 1 -- 172.21.15.59:0/4024069156 learned_addr learned my addr 172.21.15.59:0/4024069156 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:56.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.879+0000 7f1490a7a640 1 -- 172.21.15.59:0/4024069156 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275915689 0 0) 0x7f148402c1c0 con 0x7f14840c9600 2024-03-29T11:20:56.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.879+0000 7f1490a7a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:56.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.879+0000 7f1490a7a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:56.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.879+0000 7f1490a7a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:56.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.879+0000 7f1490a7a640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:20:56.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.879+0000 7f1490a7a640 1 -- 172.21.15.59:0/4024069156 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1478003520 con 0x7f14840c9600 2024-03-29T11:20:56.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.879+0000 7f1490a7a640 1 -- 172.21.15.59:0/4024069156 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432257971 0 0) 0x7f1478003520 con 0x7f14840c9600 2024-03-29T11:20:56.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.879+0000 7f1490a7a640 1 -- 172.21.15.59:0/4024069156 >> v1:172.21.15.59:6790/0 conn(0x7f1484037210 legacy=0x7f14840c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:56.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.879+0000 7f1490a7a640 1 -- 172.21.15.59:0/4024069156 >> v1:172.21.15.172:6789/0 conn(0x7f14840340d0 legacy=0x7f14840cbd60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 1 monclient: found mon.noname-a 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 1 -- 172.21.15.59:0/4024069156 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148402ac90 con 0x7f14840c9600 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 10 monclient: _finish_auth 0 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 1 -- 172.21.15.59:0/4024069156 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f14880016c0 con 0x7f14840c9600 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:56.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1490a7a640 10 monclient: dump: 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1492a7e640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:20:56.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.880+0000 7f1492a7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:56.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.881+0000 7f1490a7a640 1 -- 172.21.15.59:0/4024069156 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1488001920 con 0x7f14840c9600 2024-03-29T11:20:56.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.881+0000 7f1490a7a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:56.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.881+0000 7f1492a7e640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:56.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.881+0000 7f1492a7e640 4 set_mon_vals no callback set 2024-03-29T11:20:56.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.882+0000 7f1492a7e640 10 monclient: shutdown 2024-03-29T11:20:56.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.882+0000 7f1492a7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:56.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.882+0000 7f1492a7e640 1 -- 172.21.15.59:0/4024069156 >> v1:172.21.15.59:6789/0 conn(0x7f14840c9600 legacy=0x7f14840cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.882+0000 7f1490a7a640 1 -- 172.21.15.59:0/4024069156 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1488001d70 con 0x7f14840c9600 2024-03-29T11:20:56.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.882+0000 7f1490a7a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:20:56.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.882+0000 7f1492a7e640 1 -- 172.21.15.59:0/4024069156 shutdown_connections 2024-03-29T11:20:56.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.882+0000 7f1492a7e640 1 -- 172.21.15.59:0/4024069156 >> 172.21.15.59:0/4024069156 conn(0x7f148403a550 msgr2=0x7f14840367e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:56.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.883+0000 7f1492a7e640 1 -- 172.21.15.59:0/4024069156 shutdown_connections 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.883+0000 7f1492a7e640 1 -- 172.21.15.59:0/4024069156 wait complete. 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.884+0000 7f1492a7e640 10 monclient: build_initial_monmap 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.884+0000 7f1492a7e640 10 monclient: monmap: 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:20:56.862 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:56.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.885+0000 7f1492a7e640 1 librados: starting msgr at 2024-03-29T11:20:56.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.885+0000 7f1492a7e640 1 librados: starting objecter 2024-03-29T11:20:56.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.885+0000 7f1492a7e640 1 Processor -- start 2024-03-29T11:20:56.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.885+0000 7f1492a7e640 1 -- start start 2024-03-29T11:20:56.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.885+0000 7f1492a7e640 1 librados: setting wanted keys 2024-03-29T11:20:56.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.885+0000 7f1492a7e640 1 librados: calling monclient init 2024-03-29T11:20:56.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.885+0000 7f1492a7e640 10 monclient: init 2024-03-29T11:20:56.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.886+0000 7f1492a7e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:56.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.886+0000 7f1492a7e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:56.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.886+0000 7f1492a7e640 10 monclient(hunting): picked mon.noname-c con 0x7f14840340d0 addr v1:172.21.15.59:6790/0 2024-03-29T11:20:56.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.887+0000 7f1492a7e640 10 monclient(hunting): picked mon.noname-b con 0x7f1484037210 addr v1:172.21.15.172:6789/0 2024-03-29T11:20:56.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.887+0000 7f1492a7e640 10 monclient(hunting): picked mon.noname-a con 0x7f14840c9600 addr v1:172.21.15.59:6789/0 2024-03-29T11:20:56.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.887+0000 7f1492a7e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14840bf150 con 0x7f14840c9600 2024-03-29T11:20:56.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.887+0000 7f1492a7e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14840bb5a0 con 0x7f1484037210 2024-03-29T11:20:56.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.887+0000 7f1492a7e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14840bb880 con 0x7f14840340d0 2024-03-29T11:20:56.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.887+0000 7f1492a7e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:56.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.887+0000 7f1492a7e640 10 monclient(hunting): authenticate will time out at 907.001072s 2024-03-29T11:20:56.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.888+0000 7f1491a7c640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f14840340d0 0x7f14840cc5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:37100/0 (socket says 172.21.15.59:37100) 2024-03-29T11:20:56.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.888+0000 7f1491a7c640 1 -- 172.21.15.59:0/1716564664 learned_addr learned my addr 172.21.15.59:0/1716564664 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:20:56.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.888+0000 7f14827fc640 10 client.?.objecter ms_handle_connect 0x7f14840340d0 2024-03-29T11:20:56.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.888+0000 7f14827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:56.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.889+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199990594 0 0) 0x7f14840bb880 con 0x7f14840340d0 2024-03-29T11:20:56.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.889+0000 7f14827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:56.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.889+0000 7f14827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:56.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.889+0000 7f14827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:56.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.889+0000 7f14827fc640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:20:56.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.889+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1470004eb0 con 0x7f14840340d0 2024-03-29T11:20:56.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2731046016 0 0) 0x7f1470004eb0 con 0x7f14840340d0 2024-03-29T11:20:56.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14840bb880 con 0x7f14840340d0 2024-03-29T11:20:56.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f148c04fdd0 con 0x7f14840340d0 2024-03-29T11:20:56.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:56.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:56.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 10 monclient(hunting): dump: 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 10 client.?.objecter ms_handle_connect 0x7f14840c9600 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:56.869 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3847126996 0 0) 0x7f14840bb880 con 0x7f14840340d0 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 >> v1:172.21.15.172:6789/0 conn(0x7f1484037210 legacy=0x7f14840bbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 >> v1:172.21.15.59:6789/0 conn(0x7f14840c9600 legacy=0x7f148416bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.890+0000 7f14827fc640 1 monclient: found mon.c 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14840bbb60 con 0x7f14840340d0 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f1492a7e640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f1492a7e640 10 monclient: _renew_subs 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f1492a7e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:56.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f1492a7e640 1 -- 172.21.15.59:0/1716564664 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14840ba980 con 0x7f14840340d0 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f1492a7e640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f1492a7e640 10 monclient: _renew_subs 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f1492a7e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f1492a7e640 1 -- 172.21.15.59:0/1716564664 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14840cfb60 con 0x7f14840340d0 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f1492a7e640 1 librados: init done 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f148c04f400 con 0x7f14840340d0 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f148c075710 con 0x7f14840340d0 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f14827fc640 10 monclient: dump: 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:20:56.871 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:56.872 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:20:56.872 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:56.872 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:20:56.872 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:20:56.872 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:20:56.872 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:20:56.872 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:20:56.872 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.891+0000 7f1483fff640 4 set_mon_vals no callback set 2024-03-29T11:20:56.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.893+0000 7f1492a7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:56.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.893+0000 7f1492a7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:56.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.893+0000 7f1492a7e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:56.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.893+0000 7f1492a7e640 1 -- 172.21.15.59:0/1716564664 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14840ba7e0 con 0x7f14840340d0 2024-03-29T11:20:56.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.895+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f148c067bf0 con 0x7f14840340d0 2024-03-29T11:20:56.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.896+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f148c0ff490 con 0x7f14840340d0 2024-03-29T11:20:56.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.896+0000 7f14827fc640 10 client.4262.objecter ms_dispatch 0x7f148c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:56.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.896+0000 7f14827fc640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:56.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.896+0000 7f14827fc640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:56.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.896+0000 7f14827fc640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:20:56.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.896+0000 7f149127b640 10 monclient: get_auth_request con 0x7f147006db70 auth_method 0 2024-03-29T11:20:56.874 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.897+0000 7f14827fc640 10 client.4262.objecter ms_handle_connect 0x7f147006db70 2024-03-29T11:20:56.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.902+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f148c0cad70 con 0x7f14840340d0 2024-03-29T11:20:56.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.903+0000 7f14827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:56.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:56.903+0000 7f14827fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:57.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.290+0000 7f1492a7e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.290+0000 7f1492a7e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.290+0000 7f1492a7e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:20:57.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.290+0000 7f1492a7e640 1 -- 172.21.15.59:0/1716564664 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14840ba7e0 con 0x7f14840340d0 2024-03-29T11:20:57.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.291+0000 7f14827fc640 1 -- 172.21.15.59:0/1716564664 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f14840ba7e0 con 0x7f14840340d0 2024-03-29T11:20:57.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.291+0000 7f14827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.291+0000 7f14827fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:57.268 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:20:57.268 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:57.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.294+0000 7f1492a7e640 10 librados: watch_flush enter 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.294+0000 7f1492a7e640 10 librados: watch_flush exit 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.294+0000 7f1492a7e640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.295+0000 7f1492a7e640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.295+0000 7f1492a7e640 1 -- 172.21.15.59:0/1716564664 >> v1:172.21.15.172:6816/2420742773 conn(0x7f147006db70 legacy=0x7f1470070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.295+0000 7f1492a7e640 10 monclient: shutdown 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.295+0000 7f1492a7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.295+0000 7f1492a7e640 1 -- 172.21.15.59:0/1716564664 >> v1:172.21.15.59:6790/0 conn(0x7f14840340d0 legacy=0x7f14840cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.295+0000 7f1492a7e640 1 -- 172.21.15.59:0/1716564664 shutdown_connections 2024-03-29T11:20:57.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.295+0000 7f1492a7e640 1 -- 172.21.15.59:0/1716564664 >> 172.21.15.59:0/1716564664 conn(0x7f148403a550 msgr2=0x7f1484034900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:57.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.295+0000 7f1492a7e640 1 -- 172.21.15.59:0/1716564664 shutdown_connections 2024-03-29T11:20:57.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.295+0000 7f1492a7e640 1 -- 172.21.15.59:0/1716564664 wait complete. 2024-03-29T11:20:57.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:20:57.295+0000 7f1492a7e640 1 librados: shutdown 2024-03-29T11:20:57.290 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:04.096 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:04.247 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.270+0000 7f04d7c1e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:04.247 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.270+0000 7f04d7c1e640 10 monclient: build_initial_monmap 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.270+0000 7f04d7c1e640 10 monclient: monmap: 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:21:04.271593+0000 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:21:04.271593+0000 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:04.248 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.271+0000 7f04d7c1e640 1 Processor -- start 2024-03-29T11:21:04.249 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.271+0000 7f04d7c1e640 1 -- start start 2024-03-29T11:21:04.249 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.272+0000 7f04d7c1e640 10 monclient: init 2024-03-29T11:21:04.249 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.273+0000 7f04d7c1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:04.249 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.273+0000 7f04d7c1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:04.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.276+0000 7f04d7c1e640 10 monclient(hunting): picked mon.noname-a con 0x7f04d00b0860 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:04.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.276+0000 7f04d59ba640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f04d00b0860 0x7f04d013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39447/0 (socket says 172.21.15.59:39447) 2024-03-29T11:21:04.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.276+0000 7f04d59ba640 1 -- 172.21.15.59:0/3875787844 learned_addr learned my addr 172.21.15.59:0/3875787844 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:04.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d7c1e640 10 monclient(hunting): picked mon.noname-c con 0x7f04d013ef30 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:04.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d7c1e640 10 monclient(hunting): picked mon.noname-b con 0x7f04d00af8e0 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:04.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3875787844 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d00b0c40 con 0x7f04d00b0860 2024-03-29T11:21:04.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3875787844 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d00aed10 con 0x7f04d013ef30 2024-03-29T11:21:04.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3875787844 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d00afd50 con 0x7f04d00af8e0 2024-03-29T11:21:04.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d49b8640 1 -- 172.21.15.59:0/3875787844 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559493122 0 0) 0x7f04d00b0c40 con 0x7f04d00b0860 2024-03-29T11:21:04.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d7c1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:04.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d7c1e640 10 monclient(hunting): authenticate will time out at 914.391509s 2024-03-29T11:21:04.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d49b8640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d49b8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d49b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d49b8640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:21:04.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.277+0000 7f04d49b8640 1 -- 172.21.15.59:0/3875787844 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04bc003520 con 0x7f04d00b0860 2024-03-29T11:21:04.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.278+0000 7f04d49b8640 1 -- 172.21.15.59:0/3875787844 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3234021301 0 0) 0x7f04bc003520 con 0x7f04d00b0860 2024-03-29T11:21:04.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.278+0000 7f04d49b8640 1 -- 172.21.15.59:0/3875787844 >> v1:172.21.15.59:6790/0 conn(0x7f04d00af8e0 legacy=0x7f04d0149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.278+0000 7f04d49b8640 1 -- 172.21.15.59:0/3875787844 >> v1:172.21.15.172:6789/0 conn(0x7f04d013ef30 legacy=0x7f04d00af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:04.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.280+0000 7f04d49b8640 10 monclient: _finish_hunting 0 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.281+0000 7f04d49b8640 1 monclient: found mon.noname-a 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.281+0000 7f04d49b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.281+0000 7f04d49b8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.281+0000 7f04d49b8640 1 -- 172.21.15.59:0/3875787844 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d00ab1b0 con 0x7f04d00b0860 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.281+0000 7f04d49b8640 10 monclient: _finish_auth 0 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.281+0000 7f04d49b8640 10 monclient: _check_auth_tickets 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.281+0000 7f04d49b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.281+0000 7f04d49b8640 1 -- 172.21.15.59:0/3875787844 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f04cc001690 con 0x7f04d00b0860 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.281+0000 7f04d49b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.281+0000 7f04d49b8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.281+0000 7f04d49b8640 10 monclient: dump: 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:04.258 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:04.259 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:04.259 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:04.259 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.259 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:04.259 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:04.259 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:04.259 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:04.259 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.282+0000 7f04d7c1e640 5 monclient: authenticate success, global_id 4287 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.282+0000 7f04d7c1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.283+0000 7f04d49b8640 1 -- 172.21.15.59:0/3875787844 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04cc001860 con 0x7f04d00b0860 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.283+0000 7f04d49b8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.283+0000 7f04d49b8640 1 -- 172.21.15.59:0/3875787844 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f04cc001cb0 con 0x7f04d00b0860 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.283+0000 7f04d7c1e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.283+0000 7f04d7c1e640 4 set_mon_vals no callback set 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.283+0000 7f04d49b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.284+0000 7f04d49b8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.284+0000 7f04d49b8640 10 monclient: dump: 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:04.261 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:04.262 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.262 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:04.262 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:04.262 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:04.262 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:04.262 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:04.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.284+0000 7f04d7c1e640 10 monclient: shutdown 2024-03-29T11:21:04.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.284+0000 7f04d7c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:04.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.284+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3875787844 >> v1:172.21.15.59:6789/0 conn(0x7f04d00b0860 legacy=0x7f04d013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.285+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3875787844 shutdown_connections 2024-03-29T11:21:04.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.285+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3875787844 >> 172.21.15.59:0/3875787844 conn(0x7f04d006b660 msgr2=0x7f04d006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.285+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3875787844 shutdown_connections 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.285+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3875787844 wait complete. 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.286+0000 7f04d7c1e640 10 monclient: build_initial_monmap 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.286+0000 7f04d7c1e640 10 monclient: monmap: 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:21:04.264 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.287+0000 7f04d7c1e640 1 librados: starting msgr at 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.287+0000 7f04d7c1e640 1 librados: starting objecter 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.287+0000 7f04d7c1e640 1 Processor -- start 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.287+0000 7f04d7c1e640 1 -- start start 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.287+0000 7f04d7c1e640 1 librados: setting wanted keys 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.287+0000 7f04d7c1e640 1 librados: calling monclient init 2024-03-29T11:21:04.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.287+0000 7f04d7c1e640 10 monclient: init 2024-03-29T11:21:04.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.288+0000 7f04d7c1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:04.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.288+0000 7f04d7c1e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:04.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.288+0000 7f04d7c1e640 10 monclient(hunting): picked mon.noname-c con 0x7f04d00af8e0 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:04.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.288+0000 7f04d7c1e640 10 monclient(hunting): picked mon.noname-b con 0x7f04d00b0860 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:04.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.288+0000 7f04d7c1e640 10 monclient(hunting): picked mon.noname-a con 0x7f04d013ef30 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:04.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.288+0000 7f04d7c1e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d00705f0 con 0x7f04d013ef30 2024-03-29T11:21:04.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.288+0000 7f04d7c1e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d0070860 con 0x7f04d00b0860 2024-03-29T11:21:04.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.288+0000 7f04d7c1e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d0070b40 con 0x7f04d00af8e0 2024-03-29T11:21:04.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.288+0000 7f04d7c1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:04.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.288+0000 7f04d7c1e640 10 monclient(hunting): authenticate will time out at 914.402564s 2024-03-29T11:21:04.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.291+0000 7f04d59ba640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f04d00af8e0 0x7f04d006f820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:58744/0 (socket says 172.21.15.59:58744) 2024-03-29T11:21:04.268 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.291+0000 7f04d59ba640 1 -- 172.21.15.59:0/3345670676 learned_addr learned my addr 172.21.15.59:0/3345670676 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:04.268 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.292+0000 7f04c67fc640 10 client.?.objecter ms_handle_connect 0x7f04d00af8e0 2024-03-29T11:21:04.268 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.292+0000 7f04c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:04.268 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.292+0000 7f04c67fc640 10 client.?.objecter ms_handle_connect 0x7f04d00b0860 2024-03-29T11:21:04.268 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.292+0000 7f04c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:04.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.292+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122951873 0 0) 0x7f04d0070b40 con 0x7f04d00af8e0 2024-03-29T11:21:04.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.292+0000 7f04c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.292+0000 7f04c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.292+0000 7f04c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.292+0000 7f04c67fc640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:21:04.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.292+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b4004eb0 con 0x7f04d00af8e0 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 client.?.objecter ms_handle_connect 0x7f04d013ef30 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:04.272 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854643153 0 0) 0x7f04d00705f0 con 0x7f04d013ef30 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d0070b40 con 0x7f04d013ef30 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1287470244 0 0) 0x7f04b4004eb0 con 0x7f04d00af8e0 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04d00705f0 con 0x7f04d00af8e0 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058591424 0 0) 0x7f04d0070860 con 0x7f04d00b0860 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:21:04.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b4004eb0 con 0x7f04d00b0860 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f04cc001b70 con 0x7f04d00af8e0 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): dump: 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:04.274 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2387013337 0 0) 0x7f04d0070b40 con 0x7f04d013ef30 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04d0070860 con 0x7f04d013ef30 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f04c8001bf0 con 0x7f04d013ef30 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient(hunting): dump: 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.275 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2362272513 0 0) 0x7f04d00705f0 con 0x7f04d00af8e0 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 >> v1:172.21.15.172:6789/0 conn(0x7f04d00b0860 legacy=0x7f04d00b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 >> v1:172.21.15.59:6789/0 conn(0x7f04d013ef30 legacy=0x7f04d006ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 monclient: found mon.c 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d0070e20 con 0x7f04d00af8e0 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:04.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04d7c1e640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04d7c1e640 10 monclient: _renew_subs 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04d7c1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3345670676 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d00712b0 con 0x7f04d00af8e0 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04d7c1e640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04d7c1e640 10 monclient: _renew_subs 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04d7c1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3345670676 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d014d390 con 0x7f04d00af8e0 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.293+0000 7f04d7c1e640 1 librados: init done 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.295+0000 7f04d7c1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.295+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04cc001cd0 con 0x7f04d00af8e0 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.295+0000 7f04c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.295+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f04cc004d90 con 0x7f04d00af8e0 2024-03-29T11:21:04.277 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.295+0000 7f04c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.295+0000 7f04c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.295+0000 7f04c67fc640 10 monclient: dump: 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.295+0000 7f04c77fe640 4 set_mon_vals no callback set 2024-03-29T11:21:04.278 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.296+0000 7f04d7c1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.279 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.296+0000 7f04d7c1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:04.279 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.296+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3345670676 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d0071820 con 0x7f04d00af8e0 2024-03-29T11:21:04.279 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.297+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f04cc01ad60 con 0x7f04d00af8e0 2024-03-29T11:21:04.279 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.297+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f04cc084b50 con 0x7f04d00af8e0 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.297+0000 7f04c67fc640 10 client.4268.objecter ms_dispatch 0x7f04d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.297+0000 7f04c67fc640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.297+0000 7f04c67fc640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.297+0000 7f04c67fc640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.299+0000 7f04d51b9640 10 monclient: get_auth_request con 0x7f04b406df00 auth_method 0 2024-03-29T11:21:04.280 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.301+0000 7f04c67fc640 10 client.4268.objecter ms_handle_connect 0x7f04b406df00 2024-03-29T11:21:04.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.306+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04cc050430 con 0x7f04d00af8e0 2024-03-29T11:21:04.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.306+0000 7f04c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.284 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.306+0000 7f04c67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:04.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.635+0000 7f04d7c1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:04.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.635+0000 7f04d7c1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:04.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.635+0000 7f04d7c1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:04.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.635+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3345670676 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04d006f620 con 0x7f04d00af8e0 2024-03-29T11:21:04.614 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.637+0000 7f04c67fc640 1 -- 172.21.15.59:0/3345670676 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f04cc052360 con 0x7f04d00af8e0 2024-03-29T11:21:04.614 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.638+0000 7f04c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:04.614 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.638+0000 7f04c67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:04.614 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:21:04.615 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:04.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 10 librados: watch_flush enter 2024-03-29T11:21:04.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 10 librados: watch_flush exit 2024-03-29T11:21:04.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-29T11:21:04.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 10 client.4268.objecter successfully canceled tick 2024-03-29T11:21:04.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3345670676 >> v1:172.21.15.172:6816/2420742773 conn(0x7f04b406df00 legacy=0x7f04b40703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 10 monclient: shutdown 2024-03-29T11:21:04.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:04.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3345670676 >> v1:172.21.15.59:6790/0 conn(0x7f04d00af8e0 legacy=0x7f04d006f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3345670676 shutdown_connections 2024-03-29T11:21:04.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3345670676 >> 172.21.15.59:0/3345670676 conn(0x7f04d006b660 msgr2=0x7f04d013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:04.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3345670676 shutdown_connections 2024-03-29T11:21:04.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 1 -- 172.21.15.59:0/3345670676 wait complete. 2024-03-29T11:21:04.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:04.641+0000 7f04d7c1e640 1 librados: shutdown 2024-03-29T11:21:04.633 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:11.441 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:11.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.609+0000 7f1050c88640 10 monclient: get_monmap_and_config 2024-03-29T11:21:11.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.609+0000 7f1050c88640 10 monclient: build_initial_monmap 2024-03-29T11:21:11.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.609+0000 7f1050c88640 10 monclient: monmap: 2024-03-29T11:21:11.586 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:11.586 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:11.586 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:21:11.609827+0000 2024-03-29T11:21:11.586 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:21:11.609827+0000 2024-03-29T11:21:11.586 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:11.586 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:11.586 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:11.586 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:21:11.586 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:21:11.586 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:11.587 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.610+0000 7f1050c88640 1 Processor -- start 2024-03-29T11:21:11.587 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.610+0000 7f1050c88640 1 -- start start 2024-03-29T11:21:11.587 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.610+0000 7f1050c88640 10 monclient: init 2024-03-29T11:21:11.591 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.611+0000 7f1050c88640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:11.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.611+0000 7f1050c88640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:11.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.611+0000 7f1050c88640 10 monclient(hunting): picked mon.noname-c con 0x7f104c1422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:11.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.611+0000 7f1050c88640 10 monclient(hunting): picked mon.noname-b con 0x7f104c142690 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:11.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.611+0000 7f1050c88640 10 monclient(hunting): picked mon.noname-a con 0x7f104c0b17d0 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:11.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.611+0000 7f1050c88640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104c142c80 con 0x7f104c0b17d0 2024-03-29T11:21:11.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.611+0000 7f1050c88640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104c0aae40 con 0x7f104c1422b0 2024-03-29T11:21:11.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.611+0000 7f1050c88640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104c13fbd0 con 0x7f104c142690 2024-03-29T11:21:11.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.611+0000 7f1050c88640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:11.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.611+0000 7f1050c88640 10 monclient(hunting): authenticate will time out at 921.725157s 2024-03-29T11:21:11.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.615+0000 7f104a59c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f104c1422b0 0x7f104c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:36624/0 (socket says 172.21.15.59:36624) 2024-03-29T11:21:11.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.615+0000 7f104a59c640 1 -- 172.21.15.59:0/2207845359 learned_addr learned my addr 172.21.15.59:0/2207845359 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:11.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011807347 0 0) 0x7f104c142c80 con 0x7f104c0b17d0 2024-03-29T11:21:11.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:11.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:11.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:11.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:21:11.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1034003520 con 0x7f104c0b17d0 2024-03-29T11:21:11.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585436027 0 0) 0x7f104c13fbd0 con 0x7f104c142690 2024-03-29T11:21:11.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:11.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:11.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f104c142c80 con 0x7f104c142690 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030564623 0 0) 0x7f104c0aae40 con 0x7f104c1422b0 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.616+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f104c13fbd0 con 0x7f104c1422b0 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2503818682 0 0) 0x7f1034003520 con 0x7f104c0b17d0 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 >> v1:172.21.15.59:6790/0 conn(0x7f104c142690 legacy=0x7f104c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 >> v1:172.21.15.172:6789/0 conn(0x7f104c1422b0 legacy=0x7f104c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:11.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 1 monclient: found mon.noname-a 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c151940 con 0x7f104c0b17d0 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 10 monclient: _finish_auth 0 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f1050c88640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f1050c88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1040001690 con 0x7f104c0b17d0 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.617+0000 7f104959a640 10 monclient: dump: 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:11.599 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.618+0000 7f1050c88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.618+0000 7f104959a640 1 -- 172.21.15.59:0/2207845359 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10400018b0 con 0x7f104c0b17d0 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.618+0000 7f104959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:11.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.618+0000 7f1050c88640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:11.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.618+0000 7f1050c88640 4 set_mon_vals no callback set 2024-03-29T11:21:11.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.619+0000 7f1050c88640 10 monclient: shutdown 2024-03-29T11:21:11.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.619+0000 7f1050c88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:11.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.619+0000 7f1050c88640 1 -- 172.21.15.59:0/2207845359 >> v1:172.21.15.59:6789/0 conn(0x7f104c0b17d0 legacy=0x7f104c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:11.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.619+0000 7f1050c88640 1 -- 172.21.15.59:0/2207845359 shutdown_connections 2024-03-29T11:21:11.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.619+0000 7f1050c88640 1 -- 172.21.15.59:0/2207845359 >> 172.21.15.59:0/2207845359 conn(0x7f104c066a00 msgr2=0x7f104c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:11.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.620+0000 7f1050c88640 1 -- 172.21.15.59:0/2207845359 shutdown_connections 2024-03-29T11:21:11.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.620+0000 7f1050c88640 1 -- 172.21.15.59:0/2207845359 wait complete. 2024-03-29T11:21:11.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.621+0000 7f1050c88640 10 monclient: build_initial_monmap 2024-03-29T11:21:11.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.621+0000 7f1050c88640 10 monclient: monmap: 2024-03-29T11:21:11.602 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:11.602 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:11.602 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.621+0000 7f1050c88640 1 librados: starting msgr at 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.621+0000 7f1050c88640 1 librados: starting objecter 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.621+0000 7f1050c88640 1 Processor -- start 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 1 -- start start 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 1 librados: setting wanted keys 2024-03-29T11:21:11.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 1 librados: calling monclient init 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 10 monclient: init 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 10 monclient(hunting): picked mon.noname-b con 0x7f104c0b17d0 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 10 monclient(hunting): picked mon.noname-c con 0x7f104c1422b0 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 10 monclient(hunting): picked mon.noname-a con 0x7f104c142690 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104c0711b0 con 0x7f104c142690 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104c06ea70 con 0x7f104c0b17d0 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104c06ed50 con 0x7f104c1422b0 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.622+0000 7f1050c88640 10 monclient(hunting): authenticate will time out at 921.736465s 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.626+0000 7f104a59c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f104c0b17d0 0x7f104c150730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:36788/0 (socket says 172.21.15.59:36788) 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.626+0000 7f104a59c640 1 -- 172.21.15.59:0/3386985092 learned_addr learned my addr 172.21.15.59:0/3386985092 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:11.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.626+0000 7f103affd640 10 client.?.objecter ms_handle_connect 0x7f104c0b17d0 2024-03-29T11:21:11.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.626+0000 7f103affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:11.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.626+0000 7f103affd640 10 client.?.objecter ms_handle_connect 0x7f104c142690 2024-03-29T11:21:11.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.626+0000 7f103affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:11.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039674115 0 0) 0x7f104c0711b0 con 0x7f104c142690 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f102c004eb0 con 0x7f104c142690 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 854522670 0 0) 0x7f102c004eb0 con 0x7f104c142690 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f102c005300 con 0x7f104c142690 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f10400010c0 con 0x7f104c142690 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 10 monclient(hunting): dump: 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:11.607 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:11.608 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:11.608 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:11.608 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:11.608 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:11.608 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:11.608 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:11.608 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:11.608 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:11.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.627+0000 7f103affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:11.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.628+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2605627978 0 0) 0x7f102c005300 con 0x7f104c142690 2024-03-29T11:21:11.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.628+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 >> v1:172.21.15.59:6790/0 conn(0x7f104c1422b0 legacy=0x7f104c06f1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:11.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 >> v1:172.21.15.172:6789/0 conn(0x7f104c0b17d0 legacy=0x7f104c150730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f103affd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f103affd640 1 monclient: found mon.a 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f103affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f103affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c0b61f0 con 0x7f104c142690 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f103affd640 10 monclient: _finish_auth 0 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f103affd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f103affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f1050c88640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f1050c88640 10 monclient: _renew_subs 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f1050c88640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f1050c88640 1 -- 172.21.15.59:0/3386985092 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104c071de0 con 0x7f104c142690 2024-03-29T11:21:11.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f1050c88640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f1050c88640 10 monclient: _renew_subs 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f1050c88640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f1050c88640 1 -- 172.21.15.59:0/3386985092 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f104c1d67e0 con 0x7f104c142690 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.630+0000 7f1050c88640 1 librados: init done 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.632+0000 7f1050c88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.632+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1040001a20 con 0x7f104c142690 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.632+0000 7f1050c88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.632+0000 7f1050c88640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.632+0000 7f1050c88640 1 -- 172.21.15.59:0/3386985092 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104c075f10 con 0x7f104c142690 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.632+0000 7f103affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.632+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f10400010c0 con 0x7f104c142690 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.632+0000 7f103affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.632+0000 7f103affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.632+0000 7f103affd640 10 monclient: dump: 2024-03-29T11:21:11.612 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.634+0000 7f103bfff640 4 set_mon_vals no callback set 2024-03-29T11:21:11.613 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.636+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f1040003b00 con 0x7f104c142690 2024-03-29T11:21:11.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.639+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f1040083760 con 0x7f104c142690 2024-03-29T11:21:11.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.639+0000 7f103affd640 10 client.4296.objecter ms_dispatch 0x7f104c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:11.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.639+0000 7f103affd640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:11.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.639+0000 7f103affd640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:11.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.639+0000 7f103affd640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-29T11:21:11.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.639+0000 7f104a59c640 10 monclient: get_auth_request con 0x7f102c06e070 auth_method 0 2024-03-29T11:21:11.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.640+0000 7f103affd640 10 client.4296.objecter ms_handle_connect 0x7f102c06e070 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.641+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f104004f040 con 0x7f104c142690 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.641+0000 7f103affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:11.617 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.641+0000 7f103affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:11.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.971+0000 7f1050c88640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:11.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.971+0000 7f1050c88640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:11.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.971+0000 7f1050c88640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:11.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.971+0000 7f1050c88640 1 -- 172.21.15.59:0/3386985092 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f104c0b14f0 con 0x7f104c142690 2024-03-29T11:21:11.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.972+0000 7f103affd640 1 -- 172.21.15.59:0/3386985092 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f1040050f70 con 0x7f104c142690 2024-03-29T11:21:11.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.972+0000 7f103affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:11.948 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.972+0000 7f103affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:11.949 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:21:11.949 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:11.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.976+0000 7f1050c88640 10 librados: watch_flush enter 2024-03-29T11:21:11.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.976+0000 7f1050c88640 10 librados: watch_flush exit 2024-03-29T11:21:11.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.976+0000 7f1050c88640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-29T11:21:11.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.976+0000 7f1050c88640 10 client.4296.objecter successfully canceled tick 2024-03-29T11:21:11.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.976+0000 7f1050c88640 1 -- 172.21.15.59:0/3386985092 >> v1:172.21.15.172:6816/2420742773 conn(0x7f102c06e070 legacy=0x7f102c070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:11.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.976+0000 7f1050c88640 10 monclient: shutdown 2024-03-29T11:21:11.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.976+0000 7f1050c88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:11.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.976+0000 7f1050c88640 1 -- 172.21.15.59:0/3386985092 >> v1:172.21.15.59:6789/0 conn(0x7f104c142690 legacy=0x7f104c0b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:11.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.977+0000 7f1050c88640 1 -- 172.21.15.59:0/3386985092 shutdown_connections 2024-03-29T11:21:11.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.977+0000 7f1050c88640 1 -- 172.21.15.59:0/3386985092 >> 172.21.15.59:0/3386985092 conn(0x7f104c066a00 msgr2=0x7f104c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:11.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.977+0000 7f1050c88640 1 -- 172.21.15.59:0/3386985092 shutdown_connections 2024-03-29T11:21:11.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.977+0000 7f1050c88640 1 -- 172.21.15.59:0/3386985092 wait complete. 2024-03-29T11:21:11.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:11.978+0000 7f1050c88640 1 librados: shutdown 2024-03-29T11:21:11.972 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:18.779 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:18.928 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.951+0000 7f27289ab640 10 monclient: get_monmap_and_config 2024-03-29T11:21:18.928 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.951+0000 7f27289ab640 10 monclient: build_initial_monmap 2024-03-29T11:21:18.928 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.951+0000 7f27289ab640 10 monclient: monmap: 2024-03-29T11:21:18.928 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:18.928 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:18.928 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:21:18.952448+0000 2024-03-29T11:21:18.928 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:21:18.952448+0000 2024-03-29T11:21:18.928 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:18.928 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:18.929 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:18.929 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:21:18.929 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:21:18.929 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:18.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.952+0000 7f27289ab640 1 Processor -- start 2024-03-29T11:21:18.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.952+0000 7f27289ab640 1 -- start start 2024-03-29T11:21:18.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.952+0000 7f27289ab640 10 monclient: init 2024-03-29T11:21:18.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.953+0000 7f27289ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:18.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.953+0000 7f27289ab640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:18.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.953+0000 7f27289ab640 10 monclient(hunting): picked mon.noname-b con 0x7f2724071e60 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:18.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.953+0000 7f27289ab640 10 monclient(hunting): picked mon.noname-c con 0x7f27240a9c70 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:18.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.955+0000 7f27289ab640 10 monclient(hunting): picked mon.noname-a con 0x7f272413f8a0 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:18.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.955+0000 7f27289ab640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27240aaeb0 con 0x7f272413f8a0 2024-03-29T11:21:18.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f27289ab640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2724072c50 con 0x7f27240a9c70 2024-03-29T11:21:18.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f27289ab640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2724145ac0 con 0x7f2724071e60 2024-03-29T11:21:18.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f27289ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:18.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f272259c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f27240a9c70 0x7f27241453b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:34724/0 (socket says 172.21.15.59:34724) 2024-03-29T11:21:18.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f272259c640 1 -- 172.21.15.59:0/228384461 learned_addr learned my addr 172.21.15.59:0/228384461 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:18.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f27289ab640 10 monclient(hunting): authenticate will time out at 929.069708s 2024-03-29T11:21:18.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f2721d9b640 1 -- 172.21.15.59:0/228384461 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67268574 0 0) 0x7f2724072c50 con 0x7f27240a9c70 2024-03-29T11:21:18.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f2721d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:18.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f2721d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:18.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f2721d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:18.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f2721d9b640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:21:18.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f2721d9b640 1 -- 172.21.15.59:0/228384461 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f270c003520 con 0x7f27240a9c70 2024-03-29T11:21:18.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f2721d9b640 1 -- 172.21.15.59:0/228384461 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 41474142 0 0) 0x7f270c003520 con 0x7f27240a9c70 2024-03-29T11:21:18.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.956+0000 7f2721d9b640 1 -- 172.21.15.59:0/228384461 >> v1:172.21.15.59:6790/0 conn(0x7f2724071e60 legacy=0x7f272413f190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:18.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 1 -- 172.21.15.59:0/228384461 >> v1:172.21.15.59:6789/0 conn(0x7f272413f8a0 legacy=0x7f272414d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 1 monclient: found mon.noname-c 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 1 -- 172.21.15.59:0/228384461 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2724145c50 con 0x7f27240a9c70 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 10 monclient: _finish_auth 0 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 1 -- 172.21.15.59:0/228384461 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2714001690 con 0x7f27240a9c70 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f2721d9b640 10 monclient: dump: 2024-03-29T11:21:18.936 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f27289ab640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.958+0000 7f27289ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.959+0000 7f2721d9b640 1 -- 172.21.15.59:0/228384461 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27140018f0 con 0x7f27240a9c70 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.959+0000 7f2721d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.959+0000 7f2721d9b640 1 -- 172.21.15.59:0/228384461 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2714001d00 con 0x7f27240a9c70 2024-03-29T11:21:18.937 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.959+0000 7f2721d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.959+0000 7f2721d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.959+0000 7f2721d9b640 10 monclient: dump: 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.959+0000 7f27289ab640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:18.938 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.959+0000 7f27289ab640 4 set_mon_vals no callback set 2024-03-29T11:21:18.940 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.960+0000 7f27289ab640 10 monclient: shutdown 2024-03-29T11:21:18.940 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.960+0000 7f27289ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:18.940 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.960+0000 7f27289ab640 1 -- 172.21.15.59:0/228384461 >> v1:172.21.15.172:6789/0 conn(0x7f27240a9c70 legacy=0x7f27241453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:18.940 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.961+0000 7f27289ab640 1 -- 172.21.15.59:0/228384461 shutdown_connections 2024-03-29T11:21:18.940 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.961+0000 7f27289ab640 1 -- 172.21.15.59:0/228384461 >> 172.21.15.59:0/228384461 conn(0x7f27240b3a30 msgr2=0x7f27240b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:18.940 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.961+0000 7f27289ab640 1 -- 172.21.15.59:0/228384461 shutdown_connections 2024-03-29T11:21:18.940 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.961+0000 7f27289ab640 1 -- 172.21.15.59:0/228384461 wait complete. 2024-03-29T11:21:18.940 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.962+0000 7f27289ab640 10 monclient: build_initial_monmap 2024-03-29T11:21:18.940 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.962+0000 7f27289ab640 10 monclient: monmap: 2024-03-29T11:21:18.940 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.962+0000 7f27289ab640 1 librados: starting msgr at 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.962+0000 7f27289ab640 1 librados: starting objecter 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.962+0000 7f27289ab640 1 Processor -- start 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.963+0000 7f27289ab640 1 -- start start 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.963+0000 7f27289ab640 1 librados: setting wanted keys 2024-03-29T11:21:18.941 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.963+0000 7f27289ab640 1 librados: calling monclient init 2024-03-29T11:21:18.942 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.963+0000 7f27289ab640 10 monclient: init 2024-03-29T11:21:18.942 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.965+0000 7f27289ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:18.942 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.965+0000 7f27289ab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:18.951 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.968+0000 7f27289ab640 10 monclient(hunting): picked mon.noname-c con 0x7f2724071e60 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:18.951 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.968+0000 7f27289ab640 10 monclient(hunting): picked mon.noname-a con 0x7f27240a9c70 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:18.951 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f27289ab640 10 monclient(hunting): picked mon.noname-b con 0x7f272413f8a0 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:18.951 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f272359e640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f272413f8a0 0x7f272413a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:35160/0 (socket says 172.21.15.59:35160) 2024-03-29T11:21:18.951 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f272359e640 1 -- 172.21.15.59:0/1416844196 learned_addr learned my addr 172.21.15.59:0/1416844196 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:18.951 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f270b7fe640 10 client.?.objecter ms_handle_connect 0x7f272413f8a0 2024-03-29T11:21:18.951 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f270b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:18.951 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f2722d9d640 1 --1- 172.21.15.59:0/1416844196 >> v1:172.21.15.59:6790/0 conn(0x7f2724071e60 0x7f27241db320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:39035/0 (socket says 172.21.15.59:39035) 2024-03-29T11:21:18.951 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272413fcb0 con 0x7f27240a9c70 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f270b7fe640 10 client.?.objecter ms_handle_connect 0x7f2724071e60 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f270b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272413ab90 con 0x7f272413f8a0 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272406f1c0 con 0x7f2724071e60 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.969+0000 7f27289ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f27289ab640 10 monclient(hunting): authenticate will time out at 929.083740s 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199227634 0 0) 0x7f272413ab90 con 0x7f272413f8a0 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2710004eb0 con 0x7f272413f8a0 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356466621 0 0) 0x7f272406f1c0 con 0x7f2724071e60 2024-03-29T11:21:18.952 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.970+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f272413ab90 con 0x7f2724071e60 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 947541450 0 0) 0x7f272413ab90 con 0x7f2724071e60 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f272406f1c0 con 0x7f2724071e60 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2718001860 con 0x7f2724071e60 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.953 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 10 client.?.objecter ms_handle_connect 0x7f27240a9c70 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2270866685 0 0) 0x7f2710004eb0 con 0x7f272413f8a0 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f272413ab90 con 0x7f272413f8a0 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f271c001bb0 con 0x7f272413f8a0 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:18.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.971+0000 7f270b7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248007937 0 0) 0x7f272413fcb0 con 0x7f27240a9c70 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:18.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2710004eb0 con 0x7f27240a9c70 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3931650353 0 0) 0x7f2710004eb0 con 0x7f27240a9c70 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f272413fcb0 con 0x7f27240a9c70 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f27140014b0 con 0x7f27240a9c70 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:18.956 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 522038713 0 0) 0x7f272413fcb0 con 0x7f27240a9c70 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 >> v1:172.21.15.59:6790/0 conn(0x7f2724071e60 legacy=0x7f27241db320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 >> v1:172.21.15.172:6789/0 conn(0x7f272413f8a0 legacy=0x7f272413a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 1 monclient: found mon.a 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.972+0000 7f270b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272406f4a0 con 0x7f27240a9c70 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f270b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f270b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f270b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:18.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f27289ab640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f27289ab640 10 monclient: _renew_subs 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f27289ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272406e310 con 0x7f27240a9c70 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f27289ab640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f27289ab640 10 monclient: _renew_subs 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f27289ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f272406de80 con 0x7f27240a9c70 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f27289ab640 1 librados: init done 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2714004c30 con 0x7f27240a9c70 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f270b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.973+0000 7f272159a640 4 set_mon_vals no callback set 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.974+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2714004f40 con 0x7f27240a9c70 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.975+0000 7f27289ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.975+0000 7f270b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:18.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.975+0000 7f270b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.975+0000 7f270b7fe640 10 monclient: dump: 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.975+0000 7f27289ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.975+0000 7f27289ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:18.959 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.975+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2704005080 con 0x7f27240a9c70 2024-03-29T11:21:18.960 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.975+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f271401af10 con 0x7f27240a9c70 2024-03-29T11:21:18.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.976+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f2714084ad0 con 0x7f27240a9c70 2024-03-29T11:21:18.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.976+0000 7f270b7fe640 10 client.4302.objecter ms_dispatch 0x7f2724000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:18.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.976+0000 7f270b7fe640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:18.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.976+0000 7f270b7fe640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:18.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.976+0000 7f270b7fe640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T11:21:18.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.978+0000 7f2722d9d640 10 monclient: get_auth_request con 0x7f271006e1c0 auth_method 0 2024-03-29T11:21:18.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.980+0000 7f270b7fe640 10 client.4302.objecter ms_handle_connect 0x7f271006e1c0 2024-03-29T11:21:18.962 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.986+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27140503d0 con 0x7f27240a9c70 2024-03-29T11:21:18.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.986+0000 7f270b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:18.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:18.986+0000 7f270b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:19.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.317+0000 7f27289ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:19.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.317+0000 7f27289ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:19.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.317+0000 7f27289ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:19.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.317+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2704005330 con 0x7f27240a9c70 2024-03-29T11:21:19.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.318+0000 7f270b7fe640 1 -- 172.21.15.59:0/1416844196 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f2714052300 con 0x7f27240a9c70 2024-03-29T11:21:19.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.318+0000 7f270b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:19.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.318+0000 7f270b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:19.294 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:21:19.294 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:19.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.320+0000 7f27289ab640 10 librados: watch_flush enter 2024-03-29T11:21:19.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.320+0000 7f27289ab640 10 librados: watch_flush exit 2024-03-29T11:21:19.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.320+0000 7f27289ab640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T11:21:19.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.320+0000 7f27289ab640 10 client.4302.objecter successfully canceled tick 2024-03-29T11:21:19.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.321+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 >> v1:172.21.15.172:6816/2420742773 conn(0x7f271006e1c0 legacy=0x7f2710070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.321+0000 7f27289ab640 10 monclient: shutdown 2024-03-29T11:21:19.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.321+0000 7f27289ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:19.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.321+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 >> v1:172.21.15.59:6789/0 conn(0x7f27240a9c70 legacy=0x7f2724139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.321+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 shutdown_connections 2024-03-29T11:21:19.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.321+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 >> 172.21.15.59:0/1416844196 conn(0x7f27240b3a30 msgr2=0x7f27240b13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:19.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.321+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 shutdown_connections 2024-03-29T11:21:19.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.321+0000 7f27289ab640 1 -- 172.21.15.59:0/1416844196 wait complete. 2024-03-29T11:21:19.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:19.321+0000 7f27289ab640 1 librados: shutdown 2024-03-29T11:21:19.317 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:26.124 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:26.281 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.304+0000 7fba4a20f640 10 monclient: get_monmap_and_config 2024-03-29T11:21:26.281 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.305+0000 7fba4a20f640 10 monclient: build_initial_monmap 2024-03-29T11:21:26.281 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.305+0000 7fba4a20f640 10 monclient: monmap: 2024-03-29T11:21:26.281 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:26.281 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:26.281 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:21:26.305775+0000 2024-03-29T11:21:26.281 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:21:26.305775+0000 2024-03-29T11:21:26.282 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:26.282 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:26.282 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:26.282 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:21:26.282 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:21:26.282 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:26.282 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.306+0000 7fba4a20f640 1 Processor -- start 2024-03-29T11:21:26.282 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.306+0000 7fba4a20f640 1 -- start start 2024-03-29T11:21:26.283 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.306+0000 7fba4a20f640 10 monclient: init 2024-03-29T11:21:26.283 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.306+0000 7fba4a20f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:26.283 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.306+0000 7fba4a20f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:26.283 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.306+0000 7fba4a20f640 10 monclient(hunting): picked mon.noname-a con 0x7fba440b1f60 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:26.290 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.310+0000 7fba4a20f640 10 monclient(hunting): picked mon.noname-b con 0x7fba44071eb0 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:26.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.310+0000 7fba4a20f640 10 monclient(hunting): picked mon.noname-c con 0x7fba44143d10 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:26.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.310+0000 7fba4a20f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4406ef80 con 0x7fba440b1f60 2024-03-29T11:21:26.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.310+0000 7fba4a20f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4406f260 con 0x7fba44143d10 2024-03-29T11:21:26.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.310+0000 7fba4a20f640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba440b40c0 con 0x7fba44071eb0 2024-03-29T11:21:26.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.310+0000 7fba4a20f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:26.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.310+0000 7fba4a20f640 10 monclient(hunting): authenticate will time out at 936.424501s 2024-03-29T11:21:26.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.311+0000 7fba42ffd640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fba44071eb0 0x7fba44146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:46555/0 (socket says 172.21.15.59:46555) 2024-03-29T11:21:26.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.311+0000 7fba42ffd640 1 -- 172.21.15.59:0/2801530592 learned_addr learned my addr 172.21.15.59:0/2801530592 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:26.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.315+0000 7fba427fc640 1 -- 172.21.15.59:0/2801530592 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582458823 0 0) 0x7fba440b40c0 con 0x7fba44071eb0 2024-03-29T11:21:26.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.315+0000 7fba427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.315+0000 7fba427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.315+0000 7fba427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.315+0000 7fba427fc640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:21:26.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.315+0000 7fba427fc640 1 -- 172.21.15.59:0/2801530592 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba2c003520 con 0x7fba44071eb0 2024-03-29T11:21:26.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.316+0000 7fba427fc640 1 -- 172.21.15.59:0/2801530592 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2130173135 0 0) 0x7fba2c003520 con 0x7fba44071eb0 2024-03-29T11:21:26.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.316+0000 7fba427fc640 1 -- 172.21.15.59:0/2801530592 >> v1:172.21.15.172:6789/0 conn(0x7fba44143d10 legacy=0x7fba44146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 1 -- 172.21.15.59:0/2801530592 >> v1:172.21.15.59:6789/0 conn(0x7fba440b1f60 legacy=0x7fba44143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 1 monclient: found mon.noname-b 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 1 -- 172.21.15.59:0/2801530592 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba44155d70 con 0x7fba44071eb0 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 1 -- 172.21.15.59:0/2801530592 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fba34001690 con 0x7fba44071eb0 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 10 monclient: dump: 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba4a20f640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba4a20f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:26.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 1 -- 172.21.15.59:0/2801530592 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba34001860 con 0x7fba44071eb0 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba4a20f640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba4a20f640 4 set_mon_vals no callback set 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 1 -- 172.21.15.59:0/2801530592 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fba34001c70 con 0x7fba44071eb0 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.319+0000 7fba427fc640 10 monclient: dump: 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:26.299 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:26.300 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:26.300 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:26.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.320+0000 7fba4a20f640 10 monclient: shutdown 2024-03-29T11:21:26.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.320+0000 7fba4a20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:26.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.320+0000 7fba4a20f640 1 -- 172.21.15.59:0/2801530592 >> v1:172.21.15.59:6790/0 conn(0x7fba44071eb0 legacy=0x7fba44146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.320+0000 7fba4a20f640 1 -- 172.21.15.59:0/2801530592 shutdown_connections 2024-03-29T11:21:26.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.320+0000 7fba4a20f640 1 -- 172.21.15.59:0/2801530592 >> 172.21.15.59:0/2801530592 conn(0x7fba44066a00 msgr2=0x7fba44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:26.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.320+0000 7fba4a20f640 1 -- 172.21.15.59:0/2801530592 shutdown_connections 2024-03-29T11:21:26.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.320+0000 7fba4a20f640 1 -- 172.21.15.59:0/2801530592 wait complete. 2024-03-29T11:21:26.301 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.321+0000 7fba4a20f640 10 monclient: build_initial_monmap 2024-03-29T11:21:26.301 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.321+0000 7fba4a20f640 10 monclient: monmap: 2024-03-29T11:21:26.301 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:26.301 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:26.301 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:21:26.301 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:21:26.302 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:26.302 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:26.302 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:26.302 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:21:26.302 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:21:26.302 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:26.302 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.321+0000 7fba4a20f640 1 librados: starting msgr at 2024-03-29T11:21:26.302 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.321+0000 7fba4a20f640 1 librados: starting objecter 2024-03-29T11:21:26.302 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.322+0000 7fba4a20f640 1 Processor -- start 2024-03-29T11:21:26.302 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.322+0000 7fba4a20f640 1 -- start start 2024-03-29T11:21:26.302 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.322+0000 7fba4a20f640 1 librados: setting wanted keys 2024-03-29T11:21:26.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.322+0000 7fba4a20f640 1 librados: calling monclient init 2024-03-29T11:21:26.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.325+0000 7fba4a20f640 10 monclient: init 2024-03-29T11:21:26.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.326+0000 7fba4a20f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:26.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.326+0000 7fba4a20f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:26.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.326+0000 7fba4a20f640 10 monclient(hunting): picked mon.noname-b con 0x7fba440b1f60 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:26.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.326+0000 7fba4a20f640 10 monclient(hunting): picked mon.noname-c con 0x7fba44143d10 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:26.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.329+0000 7fba4a20f640 10 monclient(hunting): picked mon.noname-a con 0x7fba4406fd90 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:26.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.332+0000 7fba43fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fba4406fd90 0x7fba44140250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:56723/0 (socket says 172.21.15.59:56723) 2024-03-29T11:21:26.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.332+0000 7fba43fff640 1 -- 172.21.15.59:0/1159938194 learned_addr learned my addr 172.21.15.59:0/1159938194 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:26.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba4a20f640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba44154ce0 con 0x7fba4406fd90 2024-03-29T11:21:26.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba4a20f640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba44155540 con 0x7fba440b1f60 2024-03-29T11:21:26.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba4a20f640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba441392e0 con 0x7fba44143d10 2024-03-29T11:21:26.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba4a20f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:26.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba4a20f640 10 monclient(hunting): authenticate will time out at 936.447179s 2024-03-29T11:21:26.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 client.?.objecter ms_handle_connect 0x7fba4406fd90 2024-03-29T11:21:26.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 client.?.objecter ms_handle_connect 0x7fba44143d10 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711661970 0 0) 0x7fba44154ce0 con 0x7fba4406fd90 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba30004eb0 con 0x7fba4406fd90 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041582584 0 0) 0x7fba441392e0 con 0x7fba44143d10 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.333+0000 7fba2bfff640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:21:26.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.334+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba44154ce0 con 0x7fba44143d10 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.334+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1714332768 0 0) 0x7fba44154ce0 con 0x7fba44143d10 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.334+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba441392e0 con 0x7fba44143d10 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.334+0000 7fba2bfff640 10 client.?.objecter ms_handle_connect 0x7fba440b1f60 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.334+0000 7fba2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.334+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fba34001b70 con 0x7fba44143d10 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.334+0000 7fba2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.334+0000 7fba2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.334+0000 7fba2bfff640 10 monclient(hunting): dump: 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.315 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.334+0000 7fba2bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.334+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2352426724 0 0) 0x7fba30004eb0 con 0x7fba4406fd90 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.335+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba44154ce0 con 0x7fba4406fd90 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.335+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fba3c001a50 con 0x7fba4406fd90 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.335+0000 7fba2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.335+0000 7fba2bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.335+0000 7fba2bfff640 10 monclient(hunting): dump: 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:26.316 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.335+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4147910991 0 0) 0x7fba441392e0 con 0x7fba44143d10 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.335+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 >> v1:172.21.15.172:6789/0 conn(0x7fba440b1f60 legacy=0x7fba4406f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 >> v1:172.21.15.59:6789/0 conn(0x7fba4406fd90 legacy=0x7fba44140250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba2bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba2bfff640 1 monclient: found mon.c 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba2bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba441395c0 con 0x7fba44143d10 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba2bfff640 10 monclient: _finish_auth 0 2024-03-29T11:21:26.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba2bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:26.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:26.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba4a20f640 5 monclient: authenticate success, global_id 4280 2024-03-29T11:21:26.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba4a20f640 10 monclient: _renew_subs 2024-03-29T11:21:26.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba4a20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:26.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba4a20f640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba44153a10 con 0x7fba44143d10 2024-03-29T11:21:26.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba4a20f640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:26.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba4a20f640 10 monclient: _renew_subs 2024-03-29T11:21:26.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba4a20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:26.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba4a20f640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba44140b40 con 0x7fba44143d10 2024-03-29T11:21:26.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.336+0000 7fba4a20f640 1 librados: init done 2024-03-29T11:21:26.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.338+0000 7fba29ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:26.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.338+0000 7fba29ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:26.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.338+0000 7fba29ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:26.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.338+0000 7fba29ffb640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba24005080 con 0x7fba44143d10 2024-03-29T11:21:26.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.338+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba34001010 con 0x7fba44143d10 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.339+0000 7fba2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.339+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fba34004d90 con 0x7fba44143d10 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.339+0000 7fba2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.339+0000 7fba2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.339+0000 7fba2bfff640 10 monclient: dump: 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:26.321 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:26.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.339+0000 7fba41ffb640 4 set_mon_vals no callback set 2024-03-29T11:21:26.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.339+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fba3401ad60 con 0x7fba44143d10 2024-03-29T11:21:26.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.340+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fba34084a60 con 0x7fba44143d10 2024-03-29T11:21:26.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.340+0000 7fba2bfff640 10 client.4280.objecter ms_dispatch 0x7fba44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:26.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.340+0000 7fba437fe640 10 monclient: get_auth_request con 0x7fba3006df10 auth_method 0 2024-03-29T11:21:26.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.340+0000 7fba2bfff640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:26.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.340+0000 7fba2bfff640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:26.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.340+0000 7fba2bfff640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T11:21:26.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.344+0000 7fba2bfff640 10 client.4280.objecter ms_handle_connect 0x7fba3006df10 2024-03-29T11:21:26.325 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.349+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba34050600 con 0x7fba44143d10 2024-03-29T11:21:26.325 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.349+0000 7fba2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:26.325 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.349+0000 7fba2bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:26.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.682+0000 7fba29ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:26.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.682+0000 7fba29ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:26.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.682+0000 7fba29ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:26.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.682+0000 7fba29ffb640 1 -- 172.21.15.59:0/1159938194 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba24005330 con 0x7fba44143d10 2024-03-29T11:21:26.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.683+0000 7fba2bfff640 1 -- 172.21.15.59:0/1159938194 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fba34052530 con 0x7fba44143d10 2024-03-29T11:21:26.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.683+0000 7fba2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:26.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.683+0000 7fba2bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:26.660 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:21:26.660 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:26.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.687+0000 7fba29ffb640 10 librados: watch_flush enter 2024-03-29T11:21:26.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.687+0000 7fba29ffb640 10 librados: watch_flush exit 2024-03-29T11:21:26.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.688+0000 7fba29ffb640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T11:21:26.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.688+0000 7fba29ffb640 10 client.4280.objecter successfully canceled tick 2024-03-29T11:21:26.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.688+0000 7fba29ffb640 1 -- 172.21.15.59:0/1159938194 >> v1:172.21.15.172:6816/2420742773 conn(0x7fba3006df10 legacy=0x7fba300703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.688+0000 7fba29ffb640 10 monclient: shutdown 2024-03-29T11:21:26.665 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.688+0000 7fba29ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:26.665 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.688+0000 7fba29ffb640 1 -- 172.21.15.59:0/1159938194 >> v1:172.21.15.59:6790/0 conn(0x7fba44143d10 legacy=0x7fba4413fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.665 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.688+0000 7fba29ffb640 1 -- 172.21.15.59:0/1159938194 shutdown_connections 2024-03-29T11:21:26.665 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.688+0000 7fba29ffb640 1 -- 172.21.15.59:0/1159938194 >> 172.21.15.59:0/1159938194 conn(0x7fba44066a00 msgr2=0x7fba440b4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:26.665 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.688+0000 7fba29ffb640 1 -- 172.21.15.59:0/1159938194 shutdown_connections 2024-03-29T11:21:26.665 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.688+0000 7fba29ffb640 1 -- 172.21.15.59:0/1159938194 wait complete. 2024-03-29T11:21:26.665 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:26.688+0000 7fba29ffb640 1 librados: shutdown 2024-03-29T11:21:26.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:33.490 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:33.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.664+0000 7f1209863640 10 monclient: get_monmap_and_config 2024-03-29T11:21:33.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.664+0000 7f1209863640 10 monclient: build_initial_monmap 2024-03-29T11:21:33.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.664+0000 7f1209863640 10 monclient: monmap: 2024-03-29T11:21:33.642 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:33.642 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:33.642 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:21:33.665567+0000 2024-03-29T11:21:33.643 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:21:33.665567+0000 2024-03-29T11:21:33.643 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:33.643 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:33.643 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:33.643 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:21:33.643 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:21:33.643 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:33.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.671+0000 7f1209863640 1 Processor -- start 2024-03-29T11:21:33.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.671+0000 7f1209863640 1 -- start start 2024-03-29T11:21:33.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.671+0000 7f1209863640 10 monclient: init 2024-03-29T11:21:33.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.672+0000 7f1209863640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:33.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.672+0000 7f1209863640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:33.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.672+0000 7f1209863640 10 monclient(hunting): picked mon.noname-a con 0x7f12040a9930 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:33.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.672+0000 7f1209863640 10 monclient(hunting): picked mon.noname-c con 0x7f1204149dc0 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:33.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.672+0000 7f1209863640 10 monclient(hunting): picked mon.noname-b con 0x7f120414ff40 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:33.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.672+0000 7f1209863640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12040aae60 con 0x7f12040a9930 2024-03-29T11:21:33.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.672+0000 7f1209863640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12040b4310 con 0x7f1204149dc0 2024-03-29T11:21:33.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.672+0000 7f1209863640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12040afdd0 con 0x7f120414ff40 2024-03-29T11:21:33.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.672+0000 7f1209863640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:33.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.672+0000 7f1209863640 10 monclient(hunting): authenticate will time out at 943.786000s 2024-03-29T11:21:33.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.675+0000 7f12037fe640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f120414ff40 0x7f1204152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:54597/0 (socket says 172.21.15.59:54597) 2024-03-29T11:21:33.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.675+0000 7f12037fe640 1 -- 172.21.15.59:0/516394601 learned_addr learned my addr 172.21.15.59:0/516394601 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:33.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.677+0000 7f1201ffb640 1 -- 172.21.15.59:0/516394601 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392311917 0 0) 0x7f12040afdd0 con 0x7f120414ff40 2024-03-29T11:21:33.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.677+0000 7f1201ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:33.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.677+0000 7f1201ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:33.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.677+0000 7f1201ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:33.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.677+0000 7f1201ffb640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:21:33.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.677+0000 7f1201ffb640 1 -- 172.21.15.59:0/516394601 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11ec003520 con 0x7f120414ff40 2024-03-29T11:21:33.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 1 -- 172.21.15.59:0/516394601 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2052526309 0 0) 0x7f11ec003520 con 0x7f120414ff40 2024-03-29T11:21:33.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 1 -- 172.21.15.59:0/516394601 >> v1:172.21.15.172:6789/0 conn(0x7f1204149dc0 legacy=0x7f120414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:33.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 1 -- 172.21.15.59:0/516394601 >> v1:172.21.15.59:6789/0 conn(0x7f12040a9930 legacy=0x7f1204143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:33.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:33.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 1 monclient: found mon.noname-b 2024-03-29T11:21:33.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 1 -- 172.21.15.59:0/516394601 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120414a950 con 0x7f120414ff40 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 1 -- 172.21.15.59:0/516394601 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f11f8001690 con 0x7f120414ff40 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 10 monclient: dump: 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:33.656 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:33.657 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:33.657 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:33.658 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:33.658 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:33.658 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:33.658 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:33.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1209863640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:21:33.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1209863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:33.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 1 -- 172.21.15.59:0/516394601 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11f8001860 con 0x7f120414ff40 2024-03-29T11:21:33.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:33.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 1 -- 172.21.15.59:0/516394601 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f11f8001cb0 con 0x7f120414ff40 2024-03-29T11:21:33.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:33.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1201ffb640 10 monclient: dump: 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1209863640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:33.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.679+0000 7f1209863640 4 set_mon_vals no callback set 2024-03-29T11:21:33.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.680+0000 7f1209863640 10 monclient: shutdown 2024-03-29T11:21:33.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.680+0000 7f1209863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:33.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.680+0000 7f1209863640 1 -- 172.21.15.59:0/516394601 >> v1:172.21.15.59:6790/0 conn(0x7f120414ff40 legacy=0x7f1204152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:33.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.680+0000 7f1209863640 1 -- 172.21.15.59:0/516394601 shutdown_connections 2024-03-29T11:21:33.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.680+0000 7f1209863640 1 -- 172.21.15.59:0/516394601 >> 172.21.15.59:0/516394601 conn(0x7f12040b35e0 msgr2=0x7f12040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:33.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.680+0000 7f1209863640 1 -- 172.21.15.59:0/516394601 shutdown_connections 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.681+0000 7f1209863640 1 -- 172.21.15.59:0/516394601 wait complete. 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.681+0000 7f1209863640 10 monclient: build_initial_monmap 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.681+0000 7f1209863640 10 monclient: monmap: 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.682+0000 7f1209863640 1 librados: starting msgr at 2024-03-29T11:21:33.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.682+0000 7f1209863640 1 librados: starting objecter 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.682+0000 7f1209863640 1 Processor -- start 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 1 -- start start 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 1 librados: setting wanted keys 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 1 librados: calling monclient init 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 10 monclient: init 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 10 monclient(hunting): picked mon.noname-c con 0x7f12040a9930 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 10 monclient(hunting): picked mon.noname-a con 0x7f1204149dc0 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 10 monclient(hunting): picked mon.noname-b con 0x7f120414ff40 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120413daf0 con 0x7f1204149dc0 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1204149920 con 0x7f120414ff40 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120413cf30 con 0x7f12040a9930 2024-03-29T11:21:33.664 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:33.665 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.683+0000 7f1209863640 10 monclient(hunting): authenticate will time out at 943.797507s 2024-03-29T11:21:33.665 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.688+0000 7f12027fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1204149dc0 0x7f12041e44e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40585/0 (socket says 172.21.15.59:40585) 2024-03-29T11:21:33.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.688+0000 7f12027fc640 1 -- 172.21.15.59:0/1147402232 learned_addr learned my addr 172.21.15.59:0/1147402232 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:33.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.688+0000 7f1202ffd640 1 --1- 172.21.15.59:0/1147402232 >> v1:172.21.15.59:6790/0 conn(0x7f12040a9930 0x7f120413c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:54699/0 (socket says 172.21.15.59:54699) 2024-03-29T11:21:33.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.688+0000 7f11ebfff640 10 client.?.objecter ms_handle_connect 0x7f12040a9930 2024-03-29T11:21:33.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.688+0000 7f11ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:33.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.688+0000 7f11ebfff640 10 client.?.objecter ms_handle_connect 0x7f1204149dc0 2024-03-29T11:21:33.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.688+0000 7f11ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:33.668 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.692+0000 7f11ebfff640 10 client.?.objecter ms_handle_connect 0x7f120414ff40 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.692+0000 7f11ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.692+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639117083 0 0) 0x7f120413daf0 con 0x7f1204149dc0 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.692+0000 7f11ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.692+0000 7f11ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.692+0000 7f11ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.692+0000 7f11ebfff640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.692+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f0004eb0 con 0x7f1204149dc0 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.693+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899960385 0 0) 0x7f120413cf30 con 0x7f12040a9930 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.693+0000 7f11ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.693+0000 7f11ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.693+0000 7f11ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.693+0000 7f11ebfff640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:21:33.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.693+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f120413daf0 con 0x7f12040a9930 2024-03-29T11:21:33.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.694+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563849987 0 0) 0x7f1204149920 con 0x7f120414ff40 2024-03-29T11:21:33.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.694+0000 7f11ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:33.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.694+0000 7f11ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:33.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.694+0000 7f11ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:33.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.694+0000 7f11ebfff640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:21:33.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.694+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f120413cf30 con 0x7f120414ff40 2024-03-29T11:21:33.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4077287767 0 0) 0x7f11f0004eb0 con 0x7f1204149dc0 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1204149920 con 0x7f1204149dc0 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f11fc004400 con 0x7f1204149dc0 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 10 monclient(hunting): dump: 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:33.672 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 922688731 0 0) 0x7f120413daf0 con 0x7f12040a9930 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11f0004eb0 con 0x7f12040a9930 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f11f4003f40 con 0x7f12040a9930 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 10 monclient(hunting): dump: 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:33.673 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1958465517 0 0) 0x7f1204149920 con 0x7f1204149dc0 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 >> v1:172.21.15.59:6790/0 conn(0x7f12040a9930 legacy=0x7f120413c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 >> v1:172.21.15.172:6789/0 conn(0x7f120414ff40 legacy=0x7f12040ab710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 1 monclient: found mon.a 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120413d0c0 con 0x7f1204149dc0 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 10 monclient: _finish_auth 0 2024-03-29T11:21:33.674 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:33.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f11ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:33.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f1209863640 5 monclient: authenticate success, global_id 4311 2024-03-29T11:21:33.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f1209863640 10 monclient: _renew_subs 2024-03-29T11:21:33.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f1209863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:33.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f1209863640 1 -- 172.21.15.59:0/1147402232 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120413bd10 con 0x7f1204149dc0 2024-03-29T11:21:33.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f1209863640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:33.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f1209863640 10 monclient: _renew_subs 2024-03-29T11:21:33.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f1209863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:33.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f1209863640 1 -- 172.21.15.59:0/1147402232 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12041e4dd0 con 0x7f1204149dc0 2024-03-29T11:21:33.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.695+0000 7f1209863640 1 librados: init done 2024-03-29T11:21:33.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.699+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11fc001250 con 0x7f1204149dc0 2024-03-29T11:21:33.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.699+0000 7f11ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:33.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.699+0000 7f12017fa640 4 set_mon_vals no callback set 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.700+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f11fc001620 con 0x7f1204149dc0 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.700+0000 7f11ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.700+0000 7f11ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.700+0000 7f11ebfff640 10 monclient: dump: 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:33.677 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:33.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.700+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f11fc001840 con 0x7f1204149dc0 2024-03-29T11:21:33.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.701+0000 7f1209863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:33.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.701+0000 7f1209863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:33.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.701+0000 7f1209863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:33.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.701+0000 7f1209863640 1 -- 172.21.15.59:0/1147402232 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e4005080 con 0x7f1204149dc0 2024-03-29T11:21:33.682 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.706+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f11fc084dc0 con 0x7f1204149dc0 2024-03-29T11:21:33.682 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.706+0000 7f11ebfff640 10 client.4311.objecter ms_dispatch 0x7f1204000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:33.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.706+0000 7f11ebfff640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:33.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.706+0000 7f11ebfff640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:33.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.706+0000 7f11ebfff640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T11:21:33.684 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.707+0000 7f1202ffd640 10 monclient: get_auth_request con 0x7f11f006dfa0 auth_method 0 2024-03-29T11:21:33.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.710+0000 7f11ebfff640 10 client.4311.objecter ms_handle_connect 0x7f11f006dfa0 2024-03-29T11:21:33.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.713+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11fc050760 con 0x7f1204149dc0 2024-03-29T11:21:33.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.714+0000 7f11ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:33.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:33.714+0000 7f11ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:34.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.040+0000 7f1209863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.040+0000 7f1209863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.040+0000 7f1209863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:21:34.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.040+0000 7f1209863640 1 -- 172.21.15.59:0/1147402232 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11e4005330 con 0x7f1204149dc0 2024-03-29T11:21:34.021 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:21:34.021 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:34.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.042+0000 7f11ebfff640 1 -- 172.21.15.59:0/1147402232 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f11fc052690 con 0x7f1204149dc0 2024-03-29T11:21:34.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.042+0000 7f11ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.042+0000 7f11ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:34.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.046+0000 7f1209863640 10 librados: watch_flush enter 2024-03-29T11:21:34.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.046+0000 7f1209863640 10 librados: watch_flush exit 2024-03-29T11:21:34.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.046+0000 7f1209863640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T11:21:34.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.046+0000 7f1209863640 10 client.4311.objecter successfully canceled tick 2024-03-29T11:21:34.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.046+0000 7f1209863640 1 -- 172.21.15.59:0/1147402232 >> v1:172.21.15.172:6816/2420742773 conn(0x7f11f006dfa0 legacy=0x7f11f0070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.046+0000 7f1209863640 10 monclient: shutdown 2024-03-29T11:21:34.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.046+0000 7f1209863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:34.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.046+0000 7f1209863640 1 -- 172.21.15.59:0/1147402232 >> v1:172.21.15.59:6789/0 conn(0x7f1204149dc0 legacy=0x7f12041e44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.047+0000 7f1209863640 1 -- 172.21.15.59:0/1147402232 shutdown_connections 2024-03-29T11:21:34.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.047+0000 7f1209863640 1 -- 172.21.15.59:0/1147402232 >> 172.21.15.59:0/1147402232 conn(0x7f12040b35e0 msgr2=0x7f1204127640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:34.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.047+0000 7f1209863640 1 -- 172.21.15.59:0/1147402232 shutdown_connections 2024-03-29T11:21:34.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.047+0000 7f1209863640 1 -- 172.21.15.59:0/1147402232 wait complete. 2024-03-29T11:21:34.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:34.047+0000 7f1209863640 1 librados: shutdown 2024-03-29T11:21:34.041 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:40.848 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:40.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.019+0000 7f546c33d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:40.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.019+0000 7f546c33d640 10 monclient: build_initial_monmap 2024-03-29T11:21:40.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.019+0000 7f546c33d640 10 monclient: monmap: 2024-03-29T11:21:40.997 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:40.997 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:40.998 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:21:41.020893+0000 2024-03-29T11:21:40.998 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:21:41.020893+0000 2024-03-29T11:21:40.998 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:40.998 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:40.998 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:40.998 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:21:40.998 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:21:40.998 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:40.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.020+0000 7f546c33d640 1 Processor -- start 2024-03-29T11:21:40.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.020+0000 7f546c33d640 1 -- start start 2024-03-29T11:21:40.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.020+0000 7f546c33d640 10 monclient: init 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.021+0000 7f546c33d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.021+0000 7f546c33d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.021+0000 7f546c33d640 10 monclient(hunting): picked mon.noname-a con 0x7f54640acf60 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.021+0000 7f546c33d640 10 monclient(hunting): picked mon.noname-c con 0x7f546413ff40 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.021+0000 7f546c33d640 10 monclient(hunting): picked mon.noname-b con 0x7f54641403f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.021+0000 7f546c33d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54640b4310 con 0x7f54640acf60 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.021+0000 7f546c33d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5464140800 con 0x7f546413ff40 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.021+0000 7f546c33d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54640aec00 con 0x7f54641403f0 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.021+0000 7f546c33d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.021+0000 7f546c33d640 10 monclient(hunting): authenticate will time out at 951.136154s 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.024+0000 7f546a8da640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f54641403f0 0x7f546414bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:35067/0 (socket says 172.21.15.59:35067) 2024-03-29T11:21:41.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.024+0000 7f546a8da640 1 -- 172.21.15.59:0/1663164848 learned_addr learned my addr 172.21.15.59:0/1663164848 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:41.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 1 -- 172.21.15.59:0/1663164848 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495415975 0 0) 0x7f54640b4310 con 0x7f54640acf60 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 1 -- 172.21.15.59:0/1663164848 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5450003520 con 0x7f54640acf60 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 1 -- 172.21.15.59:0/1663164848 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679211222 0 0) 0x7f54640aec00 con 0x7f54641403f0 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.028+0000 7f54690d7640 1 -- 172.21.15.59:0/1663164848 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54640b4310 con 0x7f54641403f0 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 1 -- 172.21.15.59:0/1663164848 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671502241 0 0) 0x7f5450003520 con 0x7f54640acf60 2024-03-29T11:21:41.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 1 -- 172.21.15.59:0/1663164848 >> v1:172.21.15.59:6790/0 conn(0x7f54641403f0 legacy=0x7f546414bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 1 -- 172.21.15.59:0/1663164848 >> v1:172.21.15.172:6789/0 conn(0x7f546413ff40 legacy=0x7f5464148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 10 monclient: _finish_hunting 0 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 1 monclient: found mon.noname-a 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 1 -- 172.21.15.59:0/1663164848 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54640acb00 con 0x7f54640acf60 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 10 monclient: _finish_auth 0 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 10 monclient: _check_auth_tickets 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 1 -- 172.21.15.59:0/1663164848 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5460001690 con 0x7f54640acf60 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 10 monclient: dump: 2024-03-29T11:21:41.008 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f546c33d640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f546c33d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 1 -- 172.21.15.59:0/1663164848 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54600017f0 con 0x7f54640acf60 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:41.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 1 -- 172.21.15.59:0/1663164848 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5460001bc0 con 0x7f54640acf60 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f54690d7640 10 monclient: dump: 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:41.010 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f546c33d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.029+0000 7f546c33d640 4 set_mon_vals no callback set 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.030+0000 7f546c33d640 10 monclient: shutdown 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.030+0000 7f546c33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.030+0000 7f546c33d640 1 -- 172.21.15.59:0/1663164848 >> v1:172.21.15.59:6789/0 conn(0x7f54640acf60 legacy=0x7f5464139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.032+0000 7f546c33d640 1 -- 172.21.15.59:0/1663164848 shutdown_connections 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.032+0000 7f546c33d640 1 -- 172.21.15.59:0/1663164848 >> 172.21.15.59:0/1663164848 conn(0x7f54640b35e0 msgr2=0x7f54640b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.032+0000 7f546c33d640 1 -- 172.21.15.59:0/1663164848 shutdown_connections 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.033+0000 7f546c33d640 1 -- 172.21.15.59:0/1663164848 wait complete. 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.033+0000 7f546c33d640 10 monclient: build_initial_monmap 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.033+0000 7f546c33d640 10 monclient: monmap: 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:41.012 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:41.013 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:21:41.013 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:21:41.013 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:41.013 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:41.013 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:41.013 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:21:41.013 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:21:41.013 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:41.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.034+0000 7f546c33d640 1 librados: starting msgr at 2024-03-29T11:21:41.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.034+0000 7f546c33d640 1 librados: starting objecter 2024-03-29T11:21:41.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.034+0000 7f546c33d640 1 Processor -- start 2024-03-29T11:21:41.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.034+0000 7f546c33d640 1 -- start start 2024-03-29T11:21:41.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 1 librados: setting wanted keys 2024-03-29T11:21:41.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 1 librados: calling monclient init 2024-03-29T11:21:41.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 10 monclient: init 2024-03-29T11:21:41.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:41.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:41.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 10 monclient(hunting): picked mon.noname-c con 0x7f54640acf60 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:41.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 10 monclient(hunting): picked mon.noname-a con 0x7f546413ff40 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:41.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 10 monclient(hunting): picked mon.noname-b con 0x7f54641403f0 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:41.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54641e7960 con 0x7f546413ff40 2024-03-29T11:21:41.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54641e7df0 con 0x7f54641403f0 2024-03-29T11:21:41.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546413c450 con 0x7f54640acf60 2024-03-29T11:21:41.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:41.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.035+0000 7f546c33d640 10 monclient(hunting): authenticate will time out at 951.150450s 2024-03-29T11:21:41.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.041+0000 7f54698d8640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f546413ff40 0x7f546413a060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58627/0 (socket says 172.21.15.59:58627) 2024-03-29T11:21:41.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.041+0000 7f54698d8640 1 -- 172.21.15.59:0/2273040259 learned_addr learned my addr 172.21.15.59:0/2273040259 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:41.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.042+0000 7f545affd640 10 client.?.objecter ms_handle_connect 0x7f546413ff40 2024-03-29T11:21:41.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.042+0000 7f545affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.042+0000 7f545affd640 10 client.?.objecter ms_handle_connect 0x7f54640acf60 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.042+0000 7f545affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 10 client.?.objecter ms_handle_connect 0x7f54641403f0 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180903472 0 0) 0x7f54641e7960 con 0x7f546413ff40 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5448004eb0 con 0x7f546413ff40 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896178390 0 0) 0x7f546413c450 con 0x7f54640acf60 2024-03-29T11:21:41.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:21:41.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.043+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54641e7960 con 0x7f54640acf60 2024-03-29T11:21:41.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389068010 0 0) 0x7f54641e7df0 con 0x7f54641403f0 2024-03-29T11:21:41.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:21:41.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f546413c450 con 0x7f54641403f0 2024-03-29T11:21:41.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2303736767 0 0) 0x7f54641e7960 con 0x7f54640acf60 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54641e7df0 con 0x7f54640acf60 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5460001b00 con 0x7f54640acf60 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 10 monclient(hunting): dump: 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:41.025 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:41.026 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:41.026 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.044+0000 7f545affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:41.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.046+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3977210540 0 0) 0x7f546413c450 con 0x7f54641403f0 2024-03-29T11:21:41.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.046+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54641e7960 con 0x7f54641403f0 2024-03-29T11:21:41.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.046+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5454001810 con 0x7f54641403f0 2024-03-29T11:21:41.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.046+0000 7f545affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.046+0000 7f545affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:41.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.046+0000 7f545affd640 10 monclient(hunting): dump: 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3938235693 0 0) 0x7f54641e7960 con 0x7f54641403f0 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 >> v1:172.21.15.59:6790/0 conn(0x7f54640acf60 legacy=0x7f54641375b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 >> v1:172.21.15.59:6789/0 conn(0x7f546413ff40 legacy=0x7f546413a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 1 monclient: found mon.b 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5464137cc0 con 0x7f54641403f0 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 10 monclient: _finish_auth 0 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f546c33d640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f546c33d640 10 monclient: _renew_subs 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f546c33d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f546c33d640 1 -- 172.21.15.59:0/2273040259 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5464137f30 con 0x7f54641403f0 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f546c33d640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:41.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f546c33d640 10 monclient: _renew_subs 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f546c33d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f546c33d640 1 -- 172.21.15.59:0/2273040259 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54641e73b0 con 0x7f54641403f0 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f546c33d640 1 librados: init done 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5454001c20 con 0x7f54641403f0 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f545affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.047+0000 7f54688d6640 4 set_mon_vals no callback set 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.048+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5454004980 con 0x7f54641403f0 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.048+0000 7f545affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.048+0000 7f545affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.048+0000 7f545affd640 10 monclient: dump: 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:41.033 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.049+0000 7f546c33d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.049+0000 7f546c33d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.049+0000 7f546c33d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.049+0000 7f546c33d640 1 -- 172.21.15.59:0/2273040259 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544c005080 con 0x7f54641403f0 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.050+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f545401a950 con 0x7f54641403f0 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.051+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f5454084860 con 0x7f54641403f0 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.051+0000 7f545affd640 10 client.4294.objecter ms_dispatch 0x7f5464000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:41.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.051+0000 7f545affd640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:41.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.051+0000 7f545affd640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:41.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.051+0000 7f545affd640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-29T11:21:41.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.051+0000 7f546a0d9640 10 monclient: get_auth_request con 0x7f544806de30 auth_method 0 2024-03-29T11:21:41.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.052+0000 7f545affd640 10 client.4294.objecter ms_handle_connect 0x7f544806de30 2024-03-29T11:21:41.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.061+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5454050140 con 0x7f54641403f0 2024-03-29T11:21:41.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.061+0000 7f545affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.061+0000 7f545affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:41.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.396+0000 7f546c33d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:41.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.396+0000 7f546c33d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:41.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.396+0000 7f546c33d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:21:41.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.396+0000 7f546c33d640 1 -- 172.21.15.59:0/2273040259 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f544c005330 con 0x7f54641403f0 2024-03-29T11:21:41.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.399+0000 7f545affd640 1 -- 172.21.15.59:0/2273040259 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f5454052070 con 0x7f54641403f0 2024-03-29T11:21:41.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.399+0000 7f545affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:41.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.399+0000 7f545affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:41.378 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:21:41.378 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:41.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.403+0000 7f546c33d640 10 librados: watch_flush enter 2024-03-29T11:21:41.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.403+0000 7f546c33d640 10 librados: watch_flush exit 2024-03-29T11:21:41.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.403+0000 7f546c33d640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-29T11:21:41.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.403+0000 7f546c33d640 10 client.4294.objecter successfully canceled tick 2024-03-29T11:21:41.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.403+0000 7f546c33d640 1 -- 172.21.15.59:0/2273040259 >> v1:172.21.15.172:6816/2420742773 conn(0x7f544806de30 legacy=0x7f54480702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.403+0000 7f546c33d640 10 monclient: shutdown 2024-03-29T11:21:41.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.403+0000 7f546c33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:41.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.403+0000 7f546c33d640 1 -- 172.21.15.59:0/2273040259 >> v1:172.21.15.172:6789/0 conn(0x7f54641403f0 legacy=0x7f54641ee4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.403+0000 7f546c33d640 1 -- 172.21.15.59:0/2273040259 shutdown_connections 2024-03-29T11:21:41.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.403+0000 7f546c33d640 1 -- 172.21.15.59:0/2273040259 >> 172.21.15.59:0/2273040259 conn(0x7f54640b35e0 msgr2=0x7f546413d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:41.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.404+0000 7f546c33d640 1 -- 172.21.15.59:0/2273040259 shutdown_connections 2024-03-29T11:21:41.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.406+0000 7f546c33d640 1 -- 172.21.15.59:0/2273040259 wait complete. 2024-03-29T11:21:41.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:41.406+0000 7f546c33d640 1 librados: shutdown 2024-03-29T11:21:41.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:48.206 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:48.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.384+0000 7f84bf59e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:48.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.384+0000 7f84bf59e640 10 monclient: build_initial_monmap 2024-03-29T11:21:48.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.384+0000 7f84bf59e640 10 monclient: monmap: 2024-03-29T11:21:48.362 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:48.362 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:48.362 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:21:48.385968+0000 2024-03-29T11:21:48.362 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:21:48.385968+0000 2024-03-29T11:21:48.362 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:48.362 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:48.363 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:48.363 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:21:48.363 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:21:48.363 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:48.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.385+0000 7f84bf59e640 1 Processor -- start 2024-03-29T11:21:48.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.386+0000 7f84bf59e640 1 -- start start 2024-03-29T11:21:48.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.386+0000 7f84bf59e640 10 monclient: init 2024-03-29T11:21:48.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.387+0000 7f84bf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:48.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.387+0000 7f84bf59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:48.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.387+0000 7f84bf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f84b80389e0 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:48.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.387+0000 7f84bf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f84b8038dc0 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:48.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.387+0000 7f84bf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f84b8030660 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:48.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.387+0000 7f84bf59e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b80c54c0 con 0x7f84b8038dc0 2024-03-29T11:21:48.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.387+0000 7f84bf59e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b8036430 con 0x7f84b80389e0 2024-03-29T11:21:48.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.387+0000 7f84bf59e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b8039840 con 0x7f84b8030660 2024-03-29T11:21:48.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.387+0000 7f84bf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:48.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.387+0000 7f84bf59e640 10 monclient(hunting): authenticate will time out at 958.502025s 2024-03-29T11:21:48.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.393+0000 7f84bdd9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f84b8038dc0 0x7f84b80cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:53651/0 (socket says 172.21.15.59:53651) 2024-03-29T11:21:48.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.393+0000 7f84bdd9b640 1 -- 172.21.15.59:0/621105442 learned_addr learned my addr 172.21.15.59:0/621105442 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:48.371 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 1 -- 172.21.15.59:0/621105442 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130137479 0 0) 0x7f84b80c54c0 con 0x7f84b8038dc0 2024-03-29T11:21:48.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:21:48.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 1 -- 172.21.15.59:0/621105442 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a8003520 con 0x7f84b8038dc0 2024-03-29T11:21:48.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 1 -- 172.21.15.59:0/621105442 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932527331 0 0) 0x7f84b8036430 con 0x7f84b80389e0 2024-03-29T11:21:48.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:21:48.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.394+0000 7f84bd59a640 1 -- 172.21.15.59:0/621105442 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a80048a0 con 0x7f84b80389e0 2024-03-29T11:21:48.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 1 -- 172.21.15.59:0/621105442 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1670582629 0 0) 0x7f84a80048a0 con 0x7f84b80389e0 2024-03-29T11:21:48.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 1 -- 172.21.15.59:0/621105442 >> v1:172.21.15.59:6790/0 conn(0x7f84b8030660 legacy=0x7f84b80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:48.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 1 -- 172.21.15.59:0/621105442 >> v1:172.21.15.59:6789/0 conn(0x7f84b8038dc0 legacy=0x7f84b80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:48.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 1 monclient: found mon.noname-c 2024-03-29T11:21:48.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:48.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:21:48.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 1 -- 172.21.15.59:0/621105442 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b8035690 con 0x7f84b80389e0 2024-03-29T11:21:48.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 10 monclient: _finish_auth 0 2024-03-29T11:21:48.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:48.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:48.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 1 -- 172.21.15.59:0/621105442 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f84c004f730 con 0x7f84b80389e0 2024-03-29T11:21:48.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.395+0000 7f84bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.396+0000 7f84bd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:48.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.396+0000 7f84bd59a640 10 monclient: dump: 2024-03-29T11:21:48.375 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:48.375 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:48.375 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:48.375 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:48.375 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.375 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:48.375 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:48.376 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:48.376 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:48.376 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:48.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.397+0000 7f84bf59e640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:21:48.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.397+0000 7f84bf59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:48.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.398+0000 7f84bd59a640 1 -- 172.21.15.59:0/621105442 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84c00008d0 con 0x7f84b80389e0 2024-03-29T11:21:48.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.398+0000 7f84bd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:48.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.398+0000 7f84bd59a640 1 -- 172.21.15.59:0/621105442 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f84c004f730 con 0x7f84b80389e0 2024-03-29T11:21:48.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.398+0000 7f84bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.398+0000 7f84bd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:48.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.398+0000 7f84bd59a640 10 monclient: dump: 2024-03-29T11:21:48.378 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:48.378 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:48.378 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:48.378 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:48.379 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.379 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:48.379 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:48.379 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:48.379 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:48.379 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:48.379 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.398+0000 7f84bf59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:48.379 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.398+0000 7f84bf59e640 4 set_mon_vals no callback set 2024-03-29T11:21:48.379 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.399+0000 7f84bf59e640 10 monclient: shutdown 2024-03-29T11:21:48.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.399+0000 7f84bf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:48.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.399+0000 7f84bf59e640 1 -- 172.21.15.59:0/621105442 >> v1:172.21.15.172:6789/0 conn(0x7f84b80389e0 legacy=0x7f84b8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.402+0000 7f84bf59e640 1 -- 172.21.15.59:0/621105442 shutdown_connections 2024-03-29T11:21:48.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.402+0000 7f84bf59e640 1 -- 172.21.15.59:0/621105442 >> 172.21.15.59:0/621105442 conn(0x7f84b8005c00 msgr2=0x7f84b8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:48.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.403+0000 7f84bf59e640 1 -- 172.21.15.59:0/621105442 shutdown_connections 2024-03-29T11:21:48.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.403+0000 7f84bf59e640 1 -- 172.21.15.59:0/621105442 wait complete. 2024-03-29T11:21:48.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.404+0000 7f84bf59e640 10 monclient: build_initial_monmap 2024-03-29T11:21:48.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.404+0000 7f84bf59e640 10 monclient: monmap: 2024-03-29T11:21:48.384 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:48.384 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:48.384 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:21:48.384 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:21:48.384 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:48.385 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:48.385 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:48.385 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:21:48.385 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:21:48.385 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:48.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.404+0000 7f84bf59e640 1 librados: starting msgr at 2024-03-29T11:21:48.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.404+0000 7f84bf59e640 1 librados: starting objecter 2024-03-29T11:21:48.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.404+0000 7f84bf59e640 1 Processor -- start 2024-03-29T11:21:48.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.405+0000 7f84bf59e640 1 -- start start 2024-03-29T11:21:48.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.405+0000 7f84bf59e640 1 librados: setting wanted keys 2024-03-29T11:21:48.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.405+0000 7f84bf59e640 1 librados: calling monclient init 2024-03-29T11:21:48.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.405+0000 7f84bf59e640 10 monclient: init 2024-03-29T11:21:48.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.405+0000 7f84bf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:48.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.406+0000 7f84bf59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:48.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.406+0000 7f84bf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f84b8030660 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:48.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.406+0000 7f84bf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f84b80389e0 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:48.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.406+0000 7f84bf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f84b80c5650 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:48.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.406+0000 7f84bf59e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b80c8040 con 0x7f84b80c5650 2024-03-29T11:21:48.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.406+0000 7f84bf59e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b802e700 con 0x7f84b8030660 2024-03-29T11:21:48.387 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.406+0000 7f84bf59e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b802ef30 con 0x7f84b80389e0 2024-03-29T11:21:48.387 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.406+0000 7f84bf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:48.387 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.406+0000 7f84bf59e640 10 monclient(hunting): authenticate will time out at 958.520871s 2024-03-29T11:21:48.388 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.410+0000 7f84bdd9b640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f84b80389e0 0x7f84b80c0060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:37647/0 (socket says 172.21.15.59:37647) 2024-03-29T11:21:48.388 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.411+0000 7f84bdd9b640 1 -- 172.21.15.59:0/3858232118 learned_addr learned my addr 172.21.15.59:0/3858232118 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:48.399 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.412+0000 7f84aeffd640 10 client.?.objecter ms_handle_connect 0x7f84b80389e0 2024-03-29T11:21:48.399 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.412+0000 7f84aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:48.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.412+0000 7f84aeffd640 10 client.?.objecter ms_handle_connect 0x7f84b8030660 2024-03-29T11:21:48.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.412+0000 7f84aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:48.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.412+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460734449 0 0) 0x7f84b802ef30 con 0x7f84b80389e0 2024-03-29T11:21:48.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.412+0000 7f84aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.412+0000 7f84aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.412+0000 7f84aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.412+0000 7f84aeffd640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:21:48.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.412+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a0004eb0 con 0x7f84b80389e0 2024-03-29T11:21:48.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.412+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085746145 0 0) 0x7f84b802e700 con 0x7f84b8030660 2024-03-29T11:21:48.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:21:48.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a00051d0 con 0x7f84b8030660 2024-03-29T11:21:48.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 10 client.?.objecter ms_handle_connect 0x7f84b80c5650 2024-03-29T11:21:48.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:48.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 419072633 0 0) 0x7f84a0004eb0 con 0x7f84b80389e0 2024-03-29T11:21:48.402 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84b802e700 con 0x7f84b80389e0 2024-03-29T11:21:48.402 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f84b40018d0 con 0x7f84b80389e0 2024-03-29T11:21:48.402 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.402 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:48.402 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.413+0000 7f84aeffd640 10 monclient(hunting): dump: 2024-03-29T11:21:48.402 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:48.402 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:48.402 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:48.403 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:48.403 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.403 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:48.403 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:48.403 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:48.403 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:48.403 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:48.403 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:48.403 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205082219 0 0) 0x7f84b80c8040 con 0x7f84b80c5650 2024-03-29T11:21:48.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:21:48.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a0004eb0 con 0x7f84b80c5650 2024-03-29T11:21:48.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3011744367 0 0) 0x7f84a00051d0 con 0x7f84b8030660 2024-03-29T11:21:48.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84b80c8040 con 0x7f84b8030660 2024-03-29T11:21:48.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f84c004f400 con 0x7f84b8030660 2024-03-29T11:21:48.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:48.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient(hunting): dump: 2024-03-29T11:21:48.405 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:48.405 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:48.405 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:48.405 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:48.405 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.405 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:48.406 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:48.406 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:48.406 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:48.406 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:48.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3832782187 0 0) 0x7f84b802e700 con 0x7f84b80389e0 2024-03-29T11:21:48.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 >> v1:172.21.15.172:6789/0 conn(0x7f84b8030660 legacy=0x7f84b802f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 >> v1:172.21.15.59:6789/0 conn(0x7f84b80c5650 legacy=0x7f84b80c0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:48.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 1 monclient: found mon.c 2024-03-29T11:21:48.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:48.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:48.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b80c0e80 con 0x7f84b80389e0 2024-03-29T11:21:48.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:48.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:48.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:48.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84bf59e640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:21:48.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84bf59e640 10 monclient: _renew_subs 2024-03-29T11:21:48.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84bf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:48.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84bf59e640 1 -- 172.21.15.59:0/3858232118 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b80c84d0 con 0x7f84b80389e0 2024-03-29T11:21:48.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84bf59e640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:48.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84bf59e640 10 monclient: _renew_subs 2024-03-29T11:21:48.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84bf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:48.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84bf59e640 1 -- 172.21.15.59:0/3858232118 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b802ec50 con 0x7f84b80389e0 2024-03-29T11:21:48.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.414+0000 7f84bf59e640 1 librados: init done 2024-03-29T11:21:48.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.415+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84b40047e0 con 0x7f84b80389e0 2024-03-29T11:21:48.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.415+0000 7f84aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:48.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.415+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f84b4004af0 con 0x7f84b80389e0 2024-03-29T11:21:48.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.415+0000 7f84aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.415+0000 7f84aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:48.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.415+0000 7f84aeffd640 10 monclient: dump: 2024-03-29T11:21:48.409 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:48.409 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:48.409 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:48.410 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:48.410 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.410 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:48.410 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:48.410 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:48.410 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:48.410 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:48.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.416+0000 7f84bf59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:48.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.416+0000 7f84bf59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:48.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.416+0000 7f84bf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:48.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.416+0000 7f84bf59e640 1 -- 172.21.15.59:0/3858232118 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b8007b90 con 0x7f84b80389e0 2024-03-29T11:21:48.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.416+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f84b401aac0 con 0x7f84b80389e0 2024-03-29T11:21:48.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.417+0000 7f84bcd99640 4 set_mon_vals no callback set 2024-03-29T11:21:48.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.417+0000 7f84be59c640 10 monclient: get_auth_request con 0x7f84a006e0c0 auth_method 0 2024-03-29T11:21:48.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.417+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f84b4054730 con 0x7f84b80389e0 2024-03-29T11:21:48.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.417+0000 7f84aeffd640 10 client.4295.objecter ms_dispatch 0x7f84c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:48.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.417+0000 7f84aeffd640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:48.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.417+0000 7f84aeffd640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:48.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.418+0000 7f84aeffd640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-29T11:21:48.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.418+0000 7f84aeffd640 10 client.4295.objecter ms_handle_connect 0x7f84a006e0c0 2024-03-29T11:21:48.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.426+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84b40509d0 con 0x7f84b80389e0 2024-03-29T11:21:48.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.426+0000 7f84aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:48.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.426+0000 7f84aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:48.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.759+0000 7f84bf59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:48.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.759+0000 7f84bf59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:48.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.759+0000 7f84bf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:21:48.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.759+0000 7f84bf59e640 1 -- 172.21.15.59:0/3858232118 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84b8005120 con 0x7f84b80389e0 2024-03-29T11:21:48.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.762+0000 7f84aeffd640 1 -- 172.21.15.59:0/3858232118 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f84b4050b70 con 0x7f84b80389e0 2024-03-29T11:21:48.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.762+0000 7f84aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:48.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.762+0000 7f84aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:48.739 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:21:48.739 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:48.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.765+0000 7f84bf59e640 10 librados: watch_flush enter 2024-03-29T11:21:48.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.765+0000 7f84bf59e640 10 librados: watch_flush exit 2024-03-29T11:21:48.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.765+0000 7f84bf59e640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-29T11:21:48.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.765+0000 7f84bf59e640 10 client.4295.objecter successfully canceled tick 2024-03-29T11:21:48.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.765+0000 7f84bf59e640 1 -- 172.21.15.59:0/3858232118 >> v1:172.21.15.172:6816/2420742773 conn(0x7f84a006e0c0 legacy=0x7f84a0070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.765+0000 7f84bf59e640 10 monclient: shutdown 2024-03-29T11:21:48.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.765+0000 7f84bf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:48.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.766+0000 7f84bf59e640 1 -- 172.21.15.59:0/3858232118 >> v1:172.21.15.59:6790/0 conn(0x7f84b80389e0 legacy=0x7f84b80c0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.766+0000 7f84bf59e640 1 -- 172.21.15.59:0/3858232118 shutdown_connections 2024-03-29T11:21:48.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.766+0000 7f84bf59e640 1 -- 172.21.15.59:0/3858232118 >> 172.21.15.59:0/3858232118 conn(0x7f84b8005c00 msgr2=0x7f84b80c4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:48.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.766+0000 7f84bf59e640 1 -- 172.21.15.59:0/3858232118 shutdown_connections 2024-03-29T11:21:48.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.766+0000 7f84bf59e640 1 -- 172.21.15.59:0/3858232118 wait complete. 2024-03-29T11:21:48.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:48.766+0000 7f84bf59e640 1 librados: shutdown 2024-03-29T11:21:48.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:55.568 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:55.726 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.748+0000 7f4d2d7d4640 10 monclient: get_monmap_and_config 2024-03-29T11:21:55.726 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.748+0000 7f4d2d7d4640 10 monclient: build_initial_monmap 2024-03-29T11:21:55.726 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.748+0000 7f4d2d7d4640 10 monclient: monmap: 2024-03-29T11:21:55.726 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:55.726 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:55.726 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:21:55.750620+0000 2024-03-29T11:21:55.726 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:21:55.750620+0000 2024-03-29T11:21:55.726 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:55.726 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:55.727 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:55.727 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:21:55.727 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:21:55.727 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:55.727 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.749+0000 7f4d2d7d4640 1 Processor -- start 2024-03-29T11:21:55.727 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.750+0000 7f4d2d7d4640 1 -- start start 2024-03-29T11:21:55.727 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.750+0000 7f4d2d7d4640 10 monclient: init 2024-03-29T11:21:55.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d2d7d4640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:55.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d2d7d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:55.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d2d7d4640 10 monclient(hunting): picked mon.noname-c con 0x7f4d280b0770 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:55.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d2d7d4640 10 monclient(hunting): picked mon.noname-b con 0x7f4d2814a700 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:55.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d2d7d4640 10 monclient(hunting): picked mon.noname-a con 0x7f4d280ac5a0 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:55.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d2d7d4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d280aba70 con 0x7f4d280ac5a0 2024-03-29T11:21:55.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d2d7d4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d280aecb0 con 0x7f4d280b0770 2024-03-29T11:21:55.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d2d7d4640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d280af150 con 0x7f4d2814a700 2024-03-29T11:21:55.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d2d7d4640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:55.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d2d7d4640 10 monclient(hunting): authenticate will time out at 965.866114s 2024-03-29T11:21:55.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d277fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4d280ac5a0 0x7f4d28151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40182/0 (socket says 172.21.15.59:40182) 2024-03-29T11:21:55.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.751+0000 7f4d277fe640 1 -- 172.21.15.59:0/4156122105 learned_addr learned my addr 172.21.15.59:0/4156122105 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:55.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423360769 0 0) 0x7f4d280aba70 con 0x7f4d280ac5a0 2024-03-29T11:21:55.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:55.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:55.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:55.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:21:55.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d14003520 con 0x7f4d280ac5a0 2024-03-29T11:21:55.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003491778 0 0) 0x7f4d280af150 con 0x7f4d2814a700 2024-03-29T11:21:55.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:55.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:55.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:55.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:21:55.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d280aba70 con 0x7f4d2814a700 2024-03-29T11:21:55.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.752+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017064259 0 0) 0x7f4d280aecb0 con 0x7f4d280b0770 2024-03-29T11:21:55.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:55.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:55.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:55.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:21:55.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d280af150 con 0x7f4d280b0770 2024-03-29T11:21:55.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3933744939 0 0) 0x7f4d280af150 con 0x7f4d280b0770 2024-03-29T11:21:55.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 >> v1:172.21.15.59:6790/0 conn(0x7f4d2814a700 legacy=0x7f4d280abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:55.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 >> v1:172.21.15.59:6789/0 conn(0x7f4d280ac5a0 legacy=0x7f4d28151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:55.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:55.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 1 monclient: found mon.noname-c 2024-03-29T11:21:55.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:55.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:21:55.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d280ac9d0 con 0x7f4d280b0770 2024-03-29T11:21:55.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:55.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:55.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4d18001630 con 0x7f4d280b0770 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d267fc640 10 monclient: dump: 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:55.734 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d2d7d4640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.753+0000 7f4d2d7d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.754+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d18001790 con 0x7f4d280b0770 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.754+0000 7f4d267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.754+0000 7f4d267fc640 1 -- 172.21.15.59:0/4156122105 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4d18001ba0 con 0x7f4d280b0770 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.754+0000 7f4d267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.754+0000 7f4d267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.754+0000 7f4d267fc640 10 monclient: dump: 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:55.735 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:55.736 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:55.736 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:55.736 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:55.736 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:55.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.754+0000 7f4d2d7d4640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:55.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.754+0000 7f4d2d7d4640 4 set_mon_vals no callback set 2024-03-29T11:21:55.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.755+0000 7f4d2d7d4640 10 monclient: shutdown 2024-03-29T11:21:55.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.755+0000 7f4d2d7d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:55.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.755+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/4156122105 >> v1:172.21.15.172:6789/0 conn(0x7f4d280b0770 legacy=0x7f4d28149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:55.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.755+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/4156122105 shutdown_connections 2024-03-29T11:21:55.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.755+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/4156122105 >> 172.21.15.59:0/4156122105 conn(0x7f4d2806b660 msgr2=0x7f4d2806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:55.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.755+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/4156122105 shutdown_connections 2024-03-29T11:21:55.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.755+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/4156122105 wait complete. 2024-03-29T11:21:55.742 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.756+0000 7f4d2d7d4640 10 monclient: build_initial_monmap 2024-03-29T11:21:55.742 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.756+0000 7f4d2d7d4640 10 monclient: monmap: 2024-03-29T11:21:55.742 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:21:55.742 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:55.742 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:21:55.742 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.757+0000 7f4d2d7d4640 1 librados: starting msgr at 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.757+0000 7f4d2d7d4640 1 librados: starting objecter 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.757+0000 7f4d2d7d4640 1 Processor -- start 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.757+0000 7f4d2d7d4640 1 -- start start 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.757+0000 7f4d2d7d4640 1 librados: setting wanted keys 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.757+0000 7f4d2d7d4640 1 librados: calling monclient init 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.757+0000 7f4d2d7d4640 10 monclient: init 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d2d7d4640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:55.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d2d7d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:55.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d2d7d4640 10 monclient(hunting): picked mon.noname-b con 0x7f4d280ac5a0 addr v1:172.21.15.172:6789/0 2024-03-29T11:21:55.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d2d7d4640 10 monclient(hunting): picked mon.noname-c con 0x7f4d280b0770 addr v1:172.21.15.59:6790/0 2024-03-29T11:21:55.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d2d7d4640 10 monclient(hunting): picked mon.noname-a con 0x7f4d2814a700 addr v1:172.21.15.59:6789/0 2024-03-29T11:21:55.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d2d7d4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d281285c0 con 0x7f4d2814a700 2024-03-29T11:21:55.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d2d7d4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d28125f60 con 0x7f4d280ac5a0 2024-03-29T11:21:55.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d26ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4d280ac5a0 0x7f4d281450d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:47360/0 (socket says 172.21.15.59:47360) 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d26ffd640 1 -- 172.21.15.59:0/2497868916 learned_addr learned my addr 172.21.15.59:0/2497868916 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d1f7fe640 10 client.?.objecter ms_handle_connect 0x7f4d280ac5a0 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d1f7fe640 10 client.?.objecter ms_handle_connect 0x7f4d280b0770 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d1f7fe640 10 client.?.objecter ms_handle_connect 0x7f4d2814a700 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.758+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d28145d70 con 0x7f4d280b0770 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d2d7d4640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d2d7d4640 10 monclient(hunting): authenticate will time out at 965.873757s 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421443017 0 0) 0x7f4d28125f60 con 0x7f4d280ac5a0 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:55.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d08004eb0 con 0x7f4d280ac5a0 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300063848 0 0) 0x7f4d28145d70 con 0x7f4d280b0770 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d28125f60 con 0x7f4d280b0770 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999288040 0 0) 0x7f4d281285c0 con 0x7f4d2814a700 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:21:55.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.759+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d28145d70 con 0x7f4d2814a700 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 386034700 0 0) 0x7f4d28125f60 con 0x7f4d280b0770 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d281285c0 con 0x7f4d280b0770 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4d100017b0 con 0x7f4d280b0770 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:55.748 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:55.749 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:55.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:55.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3347282662 0 0) 0x7f4d08004eb0 con 0x7f4d280ac5a0 2024-03-29T11:21:55.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d28125f60 con 0x7f4d280ac5a0 2024-03-29T11:21:55.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 795143084 0 0) 0x7f4d28145d70 con 0x7f4d2814a700 2024-03-29T11:21:55.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d08004eb0 con 0x7f4d2814a700 2024-03-29T11:21:55.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4d18001ba0 con 0x7f4d280ac5a0 2024-03-29T11:21:55.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:55.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:55.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:55.749 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4d20001b40 con 0x7f4d2814a700 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:55.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.760+0000 7f4d1f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 61225439 0 0) 0x7f4d28125f60 con 0x7f4d280ac5a0 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 >> v1:172.21.15.59:6790/0 conn(0x7f4d280b0770 legacy=0x7f4d28126760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 >> v1:172.21.15.59:6789/0 conn(0x7f4d2814a700 legacy=0x7f4d28126e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:55.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 1 monclient: found mon.b 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d281df650 con 0x7f4d280ac5a0 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d2d7d4640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d2d7d4640 10 monclient: _renew_subs 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d2d7d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d281dec10 con 0x7f4d280ac5a0 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d2d7d4640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d2d7d4640 10 monclient: _renew_subs 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d2d7d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d281457e0 con 0x7f4d280ac5a0 2024-03-29T11:21:55.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d2d7d4640 1 librados: init done 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d180038a0 con 0x7f4d280ac5a0 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4d18003c30 con 0x7f4d280ac5a0 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d1f7fe640 10 monclient: dump: 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:21:55.753 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.761+0000 7f4d257fa640 4 set_mon_vals no callback set 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.762+0000 7f4d2d7d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.762+0000 7f4d2d7d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.762+0000 7f4d2d7d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.763+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0c005080 con 0x7f4d280ac5a0 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.763+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f4d18003d90 con 0x7f4d280ac5a0 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.764+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f4d18084bc0 con 0x7f4d280ac5a0 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.764+0000 7f4d1f7fe640 10 client.4306.objecter ms_dispatch 0x7f4d28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.764+0000 7f4d1f7fe640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.764+0000 7f4d1f7fe640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.764+0000 7f4d1f7fe640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T11:21:55.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.764+0000 7f4d1ffff640 10 monclient: get_auth_request con 0x7f4d0806e220 auth_method 0 2024-03-29T11:21:55.755 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.764+0000 7f4d1f7fe640 10 client.4306.objecter ms_handle_connect 0x7f4d0806e220 2024-03-29T11:21:55.755 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.770+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d180504a0 con 0x7f4d280ac5a0 2024-03-29T11:21:55.755 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.770+0000 7f4d1f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:55.755 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:55.770+0000 7f4d1f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:56.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.184+0000 7f4d2d7d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.184+0000 7f4d2d7d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.184+0000 7f4d2d7d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:21:56.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.184+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/2497868916 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d0c005330 con 0x7f4d280ac5a0 2024-03-29T11:21:56.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.185+0000 7f4d1f7fe640 1 -- 172.21.15.59:0/2497868916 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f4d180523d0 con 0x7f4d280ac5a0 2024-03-29T11:21:56.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.185+0000 7f4d1f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.185+0000 7f4d1f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:56.163 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:21:56.163 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:56.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.189+0000 7f4d2d7d4640 10 librados: watch_flush enter 2024-03-29T11:21:56.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.189+0000 7f4d2d7d4640 10 librados: watch_flush exit 2024-03-29T11:21:56.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.190+0000 7f4d2d7d4640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T11:21:56.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.190+0000 7f4d2d7d4640 10 client.4306.objecter successfully canceled tick 2024-03-29T11:21:56.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.190+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/2497868916 >> v1:172.21.15.172:6816/2420742773 conn(0x7f4d0806e220 legacy=0x7f4d080706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.190+0000 7f4d2d7d4640 10 monclient: shutdown 2024-03-29T11:21:56.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.190+0000 7f4d2d7d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:56.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.190+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/2497868916 >> v1:172.21.15.172:6789/0 conn(0x7f4d280ac5a0 legacy=0x7f4d281450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.190+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/2497868916 shutdown_connections 2024-03-29T11:21:56.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.190+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/2497868916 >> 172.21.15.59:0/2497868916 conn(0x7f4d2806b660 msgr2=0x7f4d280abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:56.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.190+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/2497868916 shutdown_connections 2024-03-29T11:21:56.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.190+0000 7f4d2d7d4640 1 -- 172.21.15.59:0/2497868916 wait complete. 2024-03-29T11:21:56.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:21:56.191+0000 7f4d2d7d4640 1 librados: shutdown 2024-03-29T11:21:56.184 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:02.992 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:03.143 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.166+0000 7fa30961e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:03.143 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.166+0000 7fa30961e640 10 monclient: build_initial_monmap 2024-03-29T11:22:03.143 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.166+0000 7fa30961e640 10 monclient: monmap: 2024-03-29T11:22:03.143 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:03.144 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:03.144 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:22:03.167449+0000 2024-03-29T11:22:03.144 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:22:03.167449+0000 2024-03-29T11:22:03.144 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:03.144 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:03.144 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:03.144 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:22:03.144 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:22:03.144 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:03.144 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.167+0000 7fa30961e640 1 Processor -- start 2024-03-29T11:22:03.144 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.167+0000 7fa30961e640 1 -- start start 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.167+0000 7fa30961e640 10 monclient: init 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa30961e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa30961e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa30961e640 10 monclient(hunting): picked mon.noname-b con 0x7fa3040b0770 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa30961e640 10 monclient(hunting): picked mon.noname-a con 0x7fa30414a700 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa30961e640 10 monclient(hunting): picked mon.noname-c con 0x7fa3040ac5a0 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa30961e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3040aba70 con 0x7fa30414a700 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa30961e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3040aecb0 con 0x7fa3040ac5a0 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa30961e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3040af150 con 0x7fa3040b0770 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa30961e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa30961e640 10 monclient(hunting): authenticate will time out at 973.282453s 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa3027fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa30414a700 0x7fa3040abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:45018/0 (socket says 172.21.15.59:45018) 2024-03-29T11:22:03.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.168+0000 7fa3027fc640 1 -- 172.21.15.59:0/691282202 learned_addr learned my addr 172.21.15.59:0/691282202 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:03.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 1 -- 172.21.15.59:0/691282202 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437958103 0 0) 0x7fa3040aba70 con 0x7fa30414a700 2024-03-29T11:22:03.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 1 -- 172.21.15.59:0/691282202 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ec003520 con 0x7fa30414a700 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 1 -- 172.21.15.59:0/691282202 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4146235519 0 0) 0x7fa2ec003520 con 0x7fa30414a700 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 1 -- 172.21.15.59:0/691282202 >> v1:172.21.15.59:6790/0 conn(0x7fa3040b0770 legacy=0x7fa304149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 1 -- 172.21.15.59:0/691282202 >> v1:172.21.15.172:6789/0 conn(0x7fa3040ac5a0 legacy=0x7fa304151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 1 monclient: found mon.noname-a 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.169+0000 7fa301ffb640 1 -- 172.21.15.59:0/691282202 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3040ac9d0 con 0x7fa30414a700 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:03.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa30961e640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa30961e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 1 -- 172.21.15.59:0/691282202 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa2f4001a10 con 0x7fa30414a700 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 10 monclient: dump: 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:03.149 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa30961e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 1 -- 172.21.15.59:0/691282202 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2f4001c30 con 0x7fa30414a700 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa30961e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa30961e640 4 set_mon_vals no callback set 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 1 -- 172.21.15.59:0/691282202 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa2f4001fc0 con 0x7fa30414a700 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.170+0000 7fa301ffb640 10 monclient: dump: 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:03.150 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.171+0000 7fa30961e640 10 monclient: shutdown 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.171+0000 7fa30961e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.171+0000 7fa30961e640 1 -- 172.21.15.59:0/691282202 >> v1:172.21.15.59:6789/0 conn(0x7fa30414a700 legacy=0x7fa3040abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.171+0000 7fa30961e640 1 -- 172.21.15.59:0/691282202 shutdown_connections 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.171+0000 7fa30961e640 1 -- 172.21.15.59:0/691282202 >> 172.21.15.59:0/691282202 conn(0x7fa30406b660 msgr2=0x7fa30406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:03.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.171+0000 7fa30961e640 1 -- 172.21.15.59:0/691282202 shutdown_connections 2024-03-29T11:22:03.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.171+0000 7fa30961e640 1 -- 172.21.15.59:0/691282202 wait complete. 2024-03-29T11:22:03.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.172+0000 7fa30961e640 10 monclient: build_initial_monmap 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.172+0000 7fa30961e640 10 monclient: monmap: 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.172+0000 7fa30961e640 1 librados: starting msgr at 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.172+0000 7fa30961e640 1 librados: starting objecter 2024-03-29T11:22:03.155 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.172+0000 7fa30961e640 1 Processor -- start 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.173+0000 7fa30961e640 1 -- start start 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.173+0000 7fa30961e640 1 librados: setting wanted keys 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.173+0000 7fa30961e640 1 librados: calling monclient init 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.173+0000 7fa30961e640 10 monclient: init 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.173+0000 7fa30961e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.173+0000 7fa30961e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.173+0000 7fa30961e640 10 monclient(hunting): picked mon.noname-c con 0x7fa3040ac5a0 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa30961e640 10 monclient(hunting): picked mon.noname-a con 0x7fa3040b0770 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa30961e640 10 monclient(hunting): picked mon.noname-b con 0x7fa30414a700 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa30961e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3041478d0 con 0x7fa3040b0770 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa30961e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa304126f80 con 0x7fa30414a700 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa30961e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa304147650 con 0x7fa3040ac5a0 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa30961e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa30961e640 10 monclient(hunting): authenticate will time out at 973.287883s 2024-03-29T11:22:03.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa3027fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa3040b0770 0x7fa304146830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:45160/0 (socket says 172.21.15.59:45160) 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa3027fc640 1 -- 172.21.15.59:0/3036217236 learned_addr learned my addr 172.21.15.59:0/3036217236 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa2eb7fe640 10 client.?.objecter ms_handle_connect 0x7fa3040b0770 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa2eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa2eb7fe640 10 client.?.objecter ms_handle_connect 0x7fa3040ac5a0 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa2eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa2eb7fe640 10 client.?.objecter ms_handle_connect 0x7fa30414a700 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.174+0000 7fa2eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731622617 0 0) 0x7fa3041478d0 con 0x7fa3040b0770 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f0004eb0 con 0x7fa3040b0770 2024-03-29T11:22:03.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644775690 0 0) 0x7fa304126f80 con 0x7fa30414a700 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3041478d0 con 0x7fa30414a700 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469196233 0 0) 0x7fa304147650 con 0x7fa3040ac5a0 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa304126f80 con 0x7fa3040ac5a0 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1947149126 0 0) 0x7fa2f0004eb0 con 0x7fa3040b0770 2024-03-29T11:22:03.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa304147650 con 0x7fa3040b0770 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2175160607 0 0) 0x7fa304126f80 con 0x7fa3040ac5a0 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f0004eb0 con 0x7fa3040ac5a0 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa2f4001f80 con 0x7fa3040b0770 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:03.159 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa2fc001830 con 0x7fa3040ac5a0 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.175+0000 7fa2eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:03.160 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1955747270 0 0) 0x7fa3041478d0 con 0x7fa30414a700 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa304126f80 con 0x7fa30414a700 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa2f8001820 con 0x7fa30414a700 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.161 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 870696606 0 0) 0x7fa304147650 con 0x7fa3040b0770 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 >> v1:172.21.15.59:6790/0 conn(0x7fa3040ac5a0 legacy=0x7fa304124190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 >> v1:172.21.15.172:6789/0 conn(0x7fa30414a700 legacy=0x7fa304146f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 1 monclient: found mon.a 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3041239d0 con 0x7fa3040b0770 2024-03-29T11:22:03.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa30961e640 5 monclient: authenticate success, global_id 4335 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa30961e640 10 monclient: _renew_subs 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa30961e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa30961e640 1 -- 172.21.15.59:0/3036217236 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa304124900 con 0x7fa3040b0770 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa30961e640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa30961e640 10 monclient: _renew_subs 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa30961e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa30961e640 1 -- 172.21.15.59:0/3036217236 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa304124be0 con 0x7fa3040b0770 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa30961e640 1 librados: init done 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2f4004e10 con 0x7fa3040b0770 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:03.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa2f40051a0 con 0x7fa3040b0770 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.176+0000 7fa2eb7fe640 10 monclient: dump: 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.177+0000 7fa3017fa640 4 set_mon_vals no callback set 2024-03-29T11:22:03.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.178+0000 7fa30961e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.165 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.178+0000 7fa30961e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.165 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.178+0000 7fa30961e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:03.165 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.178+0000 7fa30961e640 1 -- 172.21.15.59:0/3036217236 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e4005080 con 0x7fa3040b0770 2024-03-29T11:22:03.165 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.178+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fa2f4005300 con 0x7fa3040b0770 2024-03-29T11:22:03.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.179+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fa2f4084ef0 con 0x7fa3040b0770 2024-03-29T11:22:03.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.179+0000 7fa2eb7fe640 10 client.4335.objecter ms_dispatch 0x7fa304000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:03.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.179+0000 7fa2eb7fe640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:03.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.179+0000 7fa2eb7fe640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:03.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.179+0000 7fa2eb7fe640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-29T11:22:03.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.179+0000 7fa302ffd640 10 monclient: get_auth_request con 0x7fa2f006e120 auth_method 0 2024-03-29T11:22:03.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.180+0000 7fa2eb7fe640 10 client.4335.objecter ms_handle_connect 0x7fa2f006e120 2024-03-29T11:22:03.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.189+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2f40507d0 con 0x7fa3040b0770 2024-03-29T11:22:03.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.189+0000 7fa2eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.189+0000 7fa2eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:03.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.583+0000 7fa30961e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:03.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.583+0000 7fa30961e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:03.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.583+0000 7fa30961e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:03.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.583+0000 7fa30961e640 1 -- 172.21.15.59:0/3036217236 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2e4005330 con 0x7fa3040b0770 2024-03-29T11:22:03.560 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.584+0000 7fa2eb7fe640 1 -- 172.21.15.59:0/3036217236 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fa2f4052700 con 0x7fa3040b0770 2024-03-29T11:22:03.560 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:22:03.561 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.584+0000 7fa2eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:03.561 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.584+0000 7fa2eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:03.561 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:03.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.588+0000 7fa30961e640 10 librados: watch_flush enter 2024-03-29T11:22:03.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.588+0000 7fa30961e640 10 librados: watch_flush exit 2024-03-29T11:22:03.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.588+0000 7fa30961e640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-29T11:22:03.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.588+0000 7fa30961e640 10 client.4335.objecter successfully canceled tick 2024-03-29T11:22:03.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.588+0000 7fa30961e640 1 -- 172.21.15.59:0/3036217236 >> v1:172.21.15.172:6816/2420742773 conn(0x7fa2f006e120 legacy=0x7fa2f00705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.565 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.588+0000 7fa30961e640 10 monclient: shutdown 2024-03-29T11:22:03.565 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.588+0000 7fa30961e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:03.565 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.588+0000 7fa30961e640 1 -- 172.21.15.59:0/3036217236 >> v1:172.21.15.59:6789/0 conn(0x7fa3040b0770 legacy=0x7fa304146830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.565 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.589+0000 7fa30961e640 1 -- 172.21.15.59:0/3036217236 shutdown_connections 2024-03-29T11:22:03.566 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.589+0000 7fa30961e640 1 -- 172.21.15.59:0/3036217236 >> 172.21.15.59:0/3036217236 conn(0x7fa30406b660 msgr2=0x7fa304149530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:03.566 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.589+0000 7fa30961e640 1 -- 172.21.15.59:0/3036217236 shutdown_connections 2024-03-29T11:22:03.566 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.589+0000 7fa30961e640 1 -- 172.21.15.59:0/3036217236 wait complete. 2024-03-29T11:22:03.566 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:03.590+0000 7fa30961e640 1 librados: shutdown 2024-03-29T11:22:03.580 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:10.387 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:10.529 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.553+0000 7fa1a7fff640 10 monclient: get_monmap_and_config 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.553+0000 7fa1a7fff640 10 monclient: build_initial_monmap 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.553+0000 7fa1a7fff640 10 monclient: monmap: 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:22:10.554457+0000 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:22:10.554457+0000 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:22:10.530 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:10.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.554+0000 7fa1a7fff640 1 Processor -- start 2024-03-29T11:22:10.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.554+0000 7fa1a7fff640 1 -- start start 2024-03-29T11:22:10.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.554+0000 7fa1a7fff640 10 monclient: init 2024-03-29T11:22:10.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.555+0000 7fa1a7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:10.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.555+0000 7fa1a7fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:10.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.555+0000 7fa1a7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa1a0006eb0 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:10.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.555+0000 7fa1a7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a0038f10 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:10.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.555+0000 7fa1a7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa1a0036690 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:10.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.555+0000 7fa1a7fff640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a0033c90 con 0x7fa1a0038f10 2024-03-29T11:22:10.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.555+0000 7fa1a7fff640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00389f0 con 0x7fa1a0036690 2024-03-29T11:22:10.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.555+0000 7fa1a7fff640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a0030890 con 0x7fa1a0006eb0 2024-03-29T11:22:10.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.555+0000 7fa1a7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:10.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.555+0000 7fa1a7fff640 10 monclient(hunting): authenticate will time out at 980.669243s 2024-03-29T11:22:10.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.557+0000 7fa1a6ffd640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fa1a0006eb0 0x7fa1a0031760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:47378/0 (socket says 172.21.15.59:47378) 2024-03-29T11:22:10.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.557+0000 7fa1a6ffd640 1 -- 172.21.15.59:0/1282037395 learned_addr learned my addr 172.21.15.59:0/1282037395 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:10.534 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 1 -- 172.21.15.59:0/1282037395 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645517189 0 0) 0x7fa1a0030890 con 0x7fa1a0006eb0 2024-03-29T11:22:10.534 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:10.534 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:10.534 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:10.534 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:22:10.534 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 1 -- 172.21.15.59:0/1282037395 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa190003520 con 0x7fa1a0006eb0 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 1 -- 172.21.15.59:0/1282037395 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4159835355 0 0) 0x7fa190003520 con 0x7fa1a0006eb0 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 1 -- 172.21.15.59:0/1282037395 >> v1:172.21.15.172:6789/0 conn(0x7fa1a0036690 legacy=0x7fa1a0036b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 1 -- 172.21.15.59:0/1282037395 >> v1:172.21.15.59:6789/0 conn(0x7fa1a0038f10 legacy=0x7fa1a0039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 1 -- 172.21.15.59:0/1282037395 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a00cbc40 con 0x7fa1a0006eb0 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:10.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 1 -- 172.21.15.59:0/1282037395 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa198001690 con 0x7fa1a0006eb0 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient: dump: 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a7fff640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:22:10.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 1 -- 172.21.15.59:0/1282037395 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1980018f0 con 0x7fa1a0006eb0 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 1 -- 172.21.15.59:0/1282037395 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa198001d40 con 0x7fa1a0006eb0 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a5ffb640 10 monclient: dump: 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:10.538 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:10.539 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:10.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a7fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:10.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.558+0000 7fa1a7fff640 4 set_mon_vals no callback set 2024-03-29T11:22:10.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.559+0000 7fa1a7fff640 10 monclient: shutdown 2024-03-29T11:22:10.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.559+0000 7fa1a7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:10.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.559+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1282037395 >> v1:172.21.15.59:6790/0 conn(0x7fa1a0006eb0 legacy=0x7fa1a0031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.559+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1282037395 shutdown_connections 2024-03-29T11:22:10.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.559+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1282037395 >> 172.21.15.59:0/1282037395 conn(0x7fa1a0005c20 msgr2=0x7fa1a0006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:10.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.560+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1282037395 shutdown_connections 2024-03-29T11:22:10.540 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.560+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1282037395 wait complete. 2024-03-29T11:22:10.540 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.561+0000 7fa1a7fff640 10 monclient: build_initial_monmap 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.561+0000 7fa1a7fff640 10 monclient: monmap: 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:22:10.541 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.562+0000 7fa1a7fff640 1 librados: starting msgr at 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.562+0000 7fa1a7fff640 1 librados: starting objecter 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.562+0000 7fa1a7fff640 1 Processor -- start 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.562+0000 7fa1a7fff640 1 -- start start 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.562+0000 7fa1a7fff640 1 librados: setting wanted keys 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.562+0000 7fa1a7fff640 1 librados: calling monclient init 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.562+0000 7fa1a7fff640 10 monclient: init 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.563+0000 7fa1a7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.563+0000 7fa1a7fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.563+0000 7fa1a7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa1a0006eb0 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:10.542 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.563+0000 7fa1a7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a0036690 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:10.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.563+0000 7fa1a7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa1a0038f10 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:10.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.563+0000 7fa1a7fff640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00c4470 con 0x7fa1a0036690 2024-03-29T11:22:10.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.563+0000 7fa1a7fff640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a0165760 con 0x7fa1a0006eb0 2024-03-29T11:22:10.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.563+0000 7fa1a7fff640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00c9490 con 0x7fa1a0038f10 2024-03-29T11:22:10.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.563+0000 7fa1a7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:10.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.563+0000 7fa1a7fff640 10 monclient(hunting): authenticate will time out at 980.677279s 2024-03-29T11:22:10.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.567+0000 7fa1a77fe640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fa1a0038f10 0x7fa1a0165050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:48132/0 (socket says 172.21.15.59:48132) 2024-03-29T11:22:10.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.567+0000 7fa1a77fe640 1 -- 172.21.15.59:0/1343935695 learned_addr learned my addr 172.21.15.59:0/1343935695 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:10.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.567+0000 7fa18ffff640 10 client.?.objecter ms_handle_connect 0x7fa1a0038f10 2024-03-29T11:22:10.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.567+0000 7fa18ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:10.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.567+0000 7fa18ffff640 10 client.?.objecter ms_handle_connect 0x7fa1a0036690 2024-03-29T11:22:10.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.567+0000 7fa18ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:10.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.568+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770173671 0 0) 0x7fa1a00c9490 con 0x7fa1a0038f10 2024-03-29T11:22:10.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.568+0000 7fa18ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:10.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.568+0000 7fa18ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:10.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.568+0000 7fa18ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:10.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.568+0000 7fa18ffff640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:22:10.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.568+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa194004eb0 con 0x7fa1a0038f10 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 10 client.?.objecter ms_handle_connect 0x7fa1a0006eb0 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 840197385 0 0) 0x7fa194004eb0 con 0x7fa1a0038f10 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1a00c9490 con 0x7fa1a0038f10 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa1a804fdd0 con 0x7fa1a0038f10 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 10 monclient(hunting): dump: 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:10.547 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757718640 0 0) 0x7fa1a0165760 con 0x7fa1a0006eb0 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.569+0000 7fa18ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa194004eb0 con 0x7fa1a0006eb0 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846349090 0 0) 0x7fa1a00c4470 con 0x7fa1a0036690 2024-03-29T11:22:10.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:10.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:10.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:10.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:22:10.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a0165760 con 0x7fa1a0036690 2024-03-29T11:22:10.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1242638415 0 0) 0x7fa1a0165760 con 0x7fa1a0036690 2024-03-29T11:22:10.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1a00c4470 con 0x7fa1a0036690 2024-03-29T11:22:10.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa19c001640 con 0x7fa1a0036690 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.570+0000 7fa18ffff640 10 monclient(hunting): dump: 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:10.551 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.571+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3674827129 0 0) 0x7fa194004eb0 con 0x7fa1a0006eb0 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.571+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1a0165760 con 0x7fa1a0006eb0 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.572+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3558111770 0 0) 0x7fa1a00c4470 con 0x7fa1a0036690 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.572+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 >> v1:172.21.15.59:6790/0 conn(0x7fa1a0038f10 legacy=0x7fa1a0165050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.572+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 >> v1:172.21.15.172:6789/0 conn(0x7fa1a0006eb0 legacy=0x7fa1a000c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.572+0000 7fa18ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.572+0000 7fa18ffff640 1 monclient: found mon.a 2024-03-29T11:22:10.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.572+0000 7fa18ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.573+0000 7fa18ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.573+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a000b9a0 con 0x7fa1a0036690 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.573+0000 7fa18ffff640 10 monclient: _finish_auth 0 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.573+0000 7fa18ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.573+0000 7fa18ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.573+0000 7fa1a7fff640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.573+0000 7fa1a7fff640 10 monclient: _renew_subs 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.573+0000 7fa1a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.573+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1343935695 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a000bc80 con 0x7fa1a0036690 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.574+0000 7fa1a7fff640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.574+0000 7fa1a7fff640 10 monclient: _renew_subs 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.574+0000 7fa1a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.574+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1343935695 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a000c7b0 con 0x7fa1a0036690 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.574+0000 7fa1a7fff640 1 librados: init done 2024-03-29T11:22:10.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.575+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa19c0017a0 con 0x7fa1a0036690 2024-03-29T11:22:10.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.575+0000 7fa18ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:10.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.575+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa19c0035f0 con 0x7fa1a0036690 2024-03-29T11:22:10.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.575+0000 7fa18ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:10.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.575+0000 7fa18ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:10.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.575+0000 7fa18ffff640 10 monclient: dump: 2024-03-29T11:22:10.555 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:10.555 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:10.555 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.575+0000 7fa1a57fa640 4 set_mon_vals no callback set 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.576+0000 7fa1a7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.576+0000 7fa1a7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.576+0000 7fa1a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.576+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1343935695 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a0007b70 con 0x7fa1a0036690 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.577+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fa19c003750 con 0x7fa1a0036690 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.578+0000 7fa1a6ffd640 10 monclient: get_auth_request con 0x7fa19406e010 auth_method 0 2024-03-29T11:22:10.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.578+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fa19c083510 con 0x7fa1a0036690 2024-03-29T11:22:10.557 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.578+0000 7fa18ffff640 10 client.4338.objecter ms_dispatch 0x7fa1a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:10.557 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.578+0000 7fa18ffff640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:10.557 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.578+0000 7fa18ffff640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:10.557 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.578+0000 7fa18ffff640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-29T11:22:10.557 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.578+0000 7fa18ffff640 10 client.4338.objecter ms_handle_connect 0x7fa19406e010 2024-03-29T11:22:10.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.586+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa19c04edf0 con 0x7fa1a0036690 2024-03-29T11:22:10.563 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.586+0000 7fa18ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:10.563 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.587+0000 7fa18ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:10.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.933+0000 7fa1a7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:10.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.933+0000 7fa1a7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:10.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.933+0000 7fa1a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:10.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.934+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1343935695 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1a0004cd0 con 0x7fa1a0036690 2024-03-29T11:22:10.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.935+0000 7fa18ffff640 1 -- 172.21.15.59:0/1343935695 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fa19c050d20 con 0x7fa1a0036690 2024-03-29T11:22:10.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.935+0000 7fa18ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:10.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.935+0000 7fa18ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:10.911 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:22:10.911 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.937+0000 7fa1a7fff640 10 librados: watch_flush enter 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.938+0000 7fa1a7fff640 10 librados: watch_flush exit 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.938+0000 7fa1a7fff640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.938+0000 7fa1a7fff640 10 client.4338.objecter successfully canceled tick 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.938+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1343935695 >> v1:172.21.15.172:6816/2420742773 conn(0x7fa19406e010 legacy=0x7fa1940704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.938+0000 7fa1a7fff640 10 monclient: shutdown 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.938+0000 7fa1a7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:10.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.938+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1343935695 >> v1:172.21.15.59:6789/0 conn(0x7fa1a0036690 legacy=0x7fa1a0164940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:10.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.938+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1343935695 shutdown_connections 2024-03-29T11:22:10.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.938+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1343935695 >> 172.21.15.59:0/1343935695 conn(0x7fa1a0005c20 msgr2=0x7fa1a00043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:10.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.938+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1343935695 shutdown_connections 2024-03-29T11:22:10.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.938+0000 7fa1a7fff640 1 -- 172.21.15.59:0/1343935695 wait complete. 2024-03-29T11:22:10.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:10.939+0000 7fa1a7fff640 1 librados: shutdown 2024-03-29T11:22:10.930 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:17.736 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:17.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.906+0000 7f85edcc1640 10 monclient: get_monmap_and_config 2024-03-29T11:22:17.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.906+0000 7f85edcc1640 10 monclient: build_initial_monmap 2024-03-29T11:22:17.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.906+0000 7f85edcc1640 10 monclient: monmap: 2024-03-29T11:22:17.885 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:17.885 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:22:17.907713+0000 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:22:17.907713+0000 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.908+0000 7f85edcc1640 1 Processor -- start 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.908+0000 7f85edcc1640 1 -- start start 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.908+0000 7f85edcc1640 10 monclient: init 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.908+0000 7f85edcc1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:17.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.908+0000 7f85edcc1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:17.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.908+0000 7f85edcc1640 10 monclient(hunting): picked mon.noname-c con 0x7f85e80aaee0 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:17.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.909+0000 7f85edcc1640 10 monclient(hunting): picked mon.noname-b con 0x7f85e80ae460 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:17.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.909+0000 7f85edcc1640 10 monclient(hunting): picked mon.noname-a con 0x7f85e80aeff0 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:17.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.909+0000 7f85edcc1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e80abd90 con 0x7f85e80aeff0 2024-03-29T11:22:17.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.909+0000 7f85edcc1640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e80ac070 con 0x7f85e80aaee0 2024-03-29T11:22:17.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.909+0000 7f85edcc1640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e80ac350 con 0x7f85e80ae460 2024-03-29T11:22:17.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.909+0000 7f85edcc1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:17.887 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.909+0000 7f85edcc1640 10 monclient(hunting): authenticate will time out at 988.022741s 2024-03-29T11:22:17.888 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.910+0000 7f85e7fff640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f85e80ae460 0x7f85e80ae8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:54960/0 (socket says 172.21.15.59:54960) 2024-03-29T11:22:17.888 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.910+0000 7f85e7fff640 1 -- 172.21.15.59:0/709054897 learned_addr learned my addr 172.21.15.59:0/709054897 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:17.888 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.910+0000 7f85e77fe640 1 -- 172.21.15.59:0/709054897 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209748396 0 0) 0x7f85e80ac350 con 0x7f85e80ae460 2024-03-29T11:22:17.888 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.910+0000 7f85e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:17.888 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.910+0000 7f85e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:17.888 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.910+0000 7f85e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:17.888 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.910+0000 7f85e77fe640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:22:17.888 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.910+0000 7f85e77fe640 1 -- 172.21.15.59:0/709054897 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d0003520 con 0x7f85e80ae460 2024-03-29T11:22:17.888 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.911+0000 7f85e77fe640 1 -- 172.21.15.59:0/709054897 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1599695469 0 0) 0x7f85d0003520 con 0x7f85e80ae460 2024-03-29T11:22:17.888 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.911+0000 7f85e77fe640 1 -- 172.21.15.59:0/709054897 >> v1:172.21.15.172:6789/0 conn(0x7f85e80aaee0 legacy=0x7f85e80aca50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:17.890 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 1 -- 172.21.15.59:0/709054897 >> v1:172.21.15.59:6789/0 conn(0x7f85e80aeff0 legacy=0x7f85e80af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.890 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:17.890 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 1 monclient: found mon.noname-b 2024-03-29T11:22:17.890 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:17.890 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:22:17.890 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 1 -- 172.21.15.59:0/709054897 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e80ac630 con 0x7f85e80ae460 2024-03-29T11:22:17.890 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:17.890 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:17.890 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:17.890 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 1 -- 172.21.15.59:0/709054897 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f85dc0012f0 con 0x7f85e80ae460 2024-03-29T11:22:17.891 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:17.891 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:17.891 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.914+0000 7f85e77fe640 10 monclient: dump: 2024-03-29T11:22:17.891 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:17.891 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:17.891 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:17.891 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:17.891 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:17.891 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:17.891 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:17.891 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:17.892 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:17.892 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.915+0000 7f85edcc1640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.915+0000 7f85edcc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.915+0000 7f85e77fe640 1 -- 172.21.15.59:0/709054897 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85dc0014c0 con 0x7f85e80ae460 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.915+0000 7f85e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.915+0000 7f85edcc1640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.915+0000 7f85edcc1640 4 set_mon_vals no callback set 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.916+0000 7f85edcc1640 10 monclient: shutdown 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.916+0000 7f85edcc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.916+0000 7f85edcc1640 1 -- 172.21.15.59:0/709054897 >> v1:172.21.15.59:6790/0 conn(0x7f85e80ae460 legacy=0x7f85e80ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.917+0000 7f85edcc1640 1 -- 172.21.15.59:0/709054897 shutdown_connections 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.917+0000 7f85edcc1640 1 -- 172.21.15.59:0/709054897 >> 172.21.15.59:0/709054897 conn(0x7f85e80b3a30 msgr2=0x7f85e80b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:17.894 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.918+0000 7f85edcc1640 1 -- 172.21.15.59:0/709054897 shutdown_connections 2024-03-29T11:22:17.895 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.918+0000 7f85edcc1640 1 -- 172.21.15.59:0/709054897 wait complete. 2024-03-29T11:22:17.895 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.919+0000 7f85edcc1640 10 monclient: build_initial_monmap 2024-03-29T11:22:17.895 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.919+0000 7f85edcc1640 10 monclient: monmap: 2024-03-29T11:22:17.895 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:17.895 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:17.895 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:22:17.895 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:22:17.895 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:17.895 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:17.896 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:17.896 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:22:17.896 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:22:17.896 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:17.898 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.919+0000 7f85edcc1640 1 librados: starting msgr at 2024-03-29T11:22:17.898 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.919+0000 7f85edcc1640 1 librados: starting objecter 2024-03-29T11:22:17.898 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.920+0000 7f85edcc1640 1 Processor -- start 2024-03-29T11:22:17.898 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.920+0000 7f85edcc1640 1 -- start start 2024-03-29T11:22:17.898 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.920+0000 7f85edcc1640 1 librados: setting wanted keys 2024-03-29T11:22:17.898 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.920+0000 7f85edcc1640 1 librados: calling monclient init 2024-03-29T11:22:17.898 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.920+0000 7f85edcc1640 10 monclient: init 2024-03-29T11:22:17.898 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.921+0000 7f85edcc1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:17.898 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.921+0000 7f85edcc1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:17.898 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.921+0000 7f85edcc1640 10 monclient(hunting): picked mon.noname-b con 0x7f85e80aaee0 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:17.898 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.921+0000 7f85edcc1640 10 monclient(hunting): picked mon.noname-c con 0x7f85e80ae460 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:17.899 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.921+0000 7f85edcc1640 10 monclient(hunting): picked mon.noname-a con 0x7f85e80aeff0 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:17.899 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.921+0000 7f85edcc1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e80724c0 con 0x7f85e80aeff0 2024-03-29T11:22:17.899 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.921+0000 7f85edcc1640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e81556f0 con 0x7f85e80aaee0 2024-03-29T11:22:17.899 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.921+0000 7f85edcc1640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e81559d0 con 0x7f85e80ae460 2024-03-29T11:22:17.899 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.921+0000 7f85edcc1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:17.899 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.921+0000 7f85edcc1640 10 monclient(hunting): authenticate will time out at 988.035188s 2024-03-29T11:22:17.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.925+0000 7f85e7fff640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f85e80ae460 0x7f85e808a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:57432/0 (socket says 172.21.15.59:57432) 2024-03-29T11:22:17.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.925+0000 7f85e7fff640 1 -- 172.21.15.59:0/2841500087 learned_addr learned my addr 172.21.15.59:0/2841500087 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:17.902 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.926+0000 7f85e57fa640 10 client.?.objecter ms_handle_connect 0x7f85e80aeff0 2024-03-29T11:22:17.902 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.926+0000 7f85e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:17.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.926+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007061744 0 0) 0x7f85e80724c0 con 0x7f85e80aeff0 2024-03-29T11:22:17.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.926+0000 7f85e57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:17.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.926+0000 7f85e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:17.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.926+0000 7f85e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:17.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.926+0000 7f85e57fa640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:22:17.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.926+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d4004eb0 con 0x7f85e80aeff0 2024-03-29T11:22:17.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.926+0000 7f85e57fa640 10 client.?.objecter ms_handle_connect 0x7f85e80ae460 2024-03-29T11:22:17.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.926+0000 7f85e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 10 client.?.objecter ms_handle_connect 0x7f85e80aaee0 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072055854 0 0) 0x7f85e81559d0 con 0x7f85e80ae460 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e80724c0 con 0x7f85e80ae460 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2044329614 0 0) 0x7f85d4004eb0 con 0x7f85e80aeff0 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85e81559d0 con 0x7f85e80aeff0 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f85d80016c0 con 0x7f85e80aeff0 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:17.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 10 monclient(hunting): dump: 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.927+0000 7f85e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3963730958 0 0) 0x7f85e80724c0 con 0x7f85e80ae460 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85d4004eb0 con 0x7f85e80ae460 2024-03-29T11:22:17.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686572682 0 0) 0x7f85e81556f0 con 0x7f85e80aaee0 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e80724c0 con 0x7f85e80aaee0 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f85dc001d70 con 0x7f85e80ae460 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.928+0000 7f85e57fa640 10 monclient(hunting): dump: 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:17.911 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.929+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2344368883 0 0) 0x7f85e81559d0 con 0x7f85e80aeff0 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.929+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 >> v1:172.21.15.59:6790/0 conn(0x7f85e80ae460 legacy=0x7f85e808a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.929+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 >> v1:172.21.15.172:6789/0 conn(0x7f85e80aaee0 legacy=0x7f85e81eb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.929+0000 7f85e57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.929+0000 7f85e57fa640 1 monclient: found mon.a 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.929+0000 7f85e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.929+0000 7f85e57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.929+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e808af00 con 0x7f85e80aeff0 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.929+0000 7f85e57fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:17.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.929+0000 7f85e57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.929+0000 7f85e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85edcc1640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85edcc1640 10 monclient: _renew_subs 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85edcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85edcc1640 1 -- 172.21.15.59:0/2841500087 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e8071200 con 0x7f85e80aeff0 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85d8001bd0 con 0x7f85e80aeff0 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f85d80037b0 con 0x7f85e80aeff0 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85e57fa640 10 monclient: dump: 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:17.913 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85edcc1640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85edcc1640 10 monclient: _renew_subs 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85edcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85edcc1640 1 -- 172.21.15.59:0/2841500087 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e81ebd90 con 0x7f85e80aeff0 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.930+0000 7f85edcc1640 1 librados: init done 2024-03-29T11:22:17.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.931+0000 7f85e6ffd640 4 set_mon_vals no callback set 2024-03-29T11:22:17.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.933+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f85d8019780 con 0x7f85e80aeff0 2024-03-29T11:22:17.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.933+0000 7f85edcc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:17.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.933+0000 7f85edcc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:17.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.933+0000 7f85edcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:17.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.933+0000 7f85edcc1640 1 -- 172.21.15.59:0/2841500087 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c8005080 con 0x7f85e80aeff0 2024-03-29T11:22:17.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.934+0000 7f85eccbf640 10 monclient: get_auth_request con 0x7f85d406df50 auth_method 0 2024-03-29T11:22:17.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.934+0000 7f85e57fa640 10 client.4344.objecter ms_handle_connect 0x7f85d406df50 2024-03-29T11:22:17.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.935+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f85d80534c0 con 0x7f85e80aeff0 2024-03-29T11:22:17.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.935+0000 7f85e57fa640 10 client.4344.objecter ms_dispatch 0x7f85e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.935+0000 7f85e57fa640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.935+0000 7f85e57fa640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:17.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.935+0000 7f85e57fa640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-29T11:22:17.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.942+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85d804efd0 con 0x7f85e80aeff0 2024-03-29T11:22:17.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.942+0000 7f85e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:17.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:17.942+0000 7f85e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:18.247 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.271+0000 7f85edcc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.247 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.271+0000 7f85edcc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.247 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.271+0000 7f85edcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:18.247 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.271+0000 7f85edcc1640 1 -- 172.21.15.59:0/2841500087 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85c8005330 con 0x7f85e80aeff0 2024-03-29T11:22:18.248 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.272+0000 7f85e57fa640 1 -- 172.21.15.59:0/2841500087 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f85d8050f00 con 0x7f85e80aeff0 2024-03-29T11:22:18.248 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.272+0000 7f85e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.248 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.272+0000 7f85e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:18.249 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:22:18.249 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:18.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.275+0000 7f85edcc1640 10 librados: watch_flush enter 2024-03-29T11:22:18.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.276+0000 7f85edcc1640 10 librados: watch_flush exit 2024-03-29T11:22:18.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.276+0000 7f85edcc1640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-29T11:22:18.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.276+0000 7f85edcc1640 10 client.4344.objecter successfully canceled tick 2024-03-29T11:22:18.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.277+0000 7f85edcc1640 1 -- 172.21.15.59:0/2841500087 >> v1:172.21.15.172:6816/2420742773 conn(0x7f85d406df50 legacy=0x7f85d4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.277+0000 7f85edcc1640 10 monclient: shutdown 2024-03-29T11:22:18.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.277+0000 7f85edcc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:18.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.277+0000 7f85edcc1640 1 -- 172.21.15.59:0/2841500087 >> v1:172.21.15.59:6789/0 conn(0x7f85e80aeff0 legacy=0x7f85e808a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.278+0000 7f85edcc1640 1 -- 172.21.15.59:0/2841500087 shutdown_connections 2024-03-29T11:22:18.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.278+0000 7f85edcc1640 1 -- 172.21.15.59:0/2841500087 >> 172.21.15.59:0/2841500087 conn(0x7f85e80b3a30 msgr2=0x7f85e8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:18.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.278+0000 7f85edcc1640 1 -- 172.21.15.59:0/2841500087 shutdown_connections 2024-03-29T11:22:18.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.278+0000 7f85edcc1640 1 -- 172.21.15.59:0/2841500087 wait complete. 2024-03-29T11:22:18.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:18.278+0000 7f85edcc1640 1 librados: shutdown 2024-03-29T11:22:18.271 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:25.077 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:25.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.250+0000 7fe75d647640 10 monclient: get_monmap_and_config 2024-03-29T11:22:25.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.250+0000 7fe75d647640 10 monclient: build_initial_monmap 2024-03-29T11:22:25.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.250+0000 7fe75d647640 10 monclient: monmap: 2024-03-29T11:22:25.229 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:25.230 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:25.230 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:22:25.251512+0000 2024-03-29T11:22:25.230 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:22:25.251512+0000 2024-03-29T11:22:25.230 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:25.230 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:25.230 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:25.230 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:22:25.230 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:22:25.230 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:25.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.251+0000 7fe75d647640 1 Processor -- start 2024-03-29T11:22:25.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.251+0000 7fe75d647640 1 -- start start 2024-03-29T11:22:25.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.251+0000 7fe75d647640 10 monclient: init 2024-03-29T11:22:25.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.252+0000 7fe75d647640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:25.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.252+0000 7fe75d647640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:25.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.252+0000 7fe75d647640 10 monclient(hunting): picked mon.noname-b con 0x7fe7580a9ac0 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:25.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.252+0000 7fe75d647640 10 monclient(hunting): picked mon.noname-c con 0x7fe7580b4100 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:25.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.252+0000 7fe75d647640 10 monclient(hunting): picked mon.noname-a con 0x7fe75813ee90 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:25.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.252+0000 7fe75d647640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7580af6f0 con 0x7fe75813ee90 2024-03-29T11:22:25.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.252+0000 7fe75d647640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7580aac80 con 0x7fe7580b4100 2024-03-29T11:22:25.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.252+0000 7fe75d647640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7580aaf60 con 0x7fe7580a9ac0 2024-03-29T11:22:25.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.252+0000 7fe75d647640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:25.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.252+0000 7fe75d647640 10 monclient(hunting): authenticate will time out at 995.366567s 2024-03-29T11:22:25.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.253+0000 7fe7567fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe7580b4100 0x7fe758146ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:45366/0 (socket says 172.21.15.59:45366) 2024-03-29T11:22:25.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.253+0000 7fe7567fc640 1 -- 172.21.15.59:0/3985114079 learned_addr learned my addr 172.21.15.59:0/3985114079 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:25.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.254+0000 7fe755ffb640 1 -- 172.21.15.59:0/3985114079 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039369063 0 0) 0x7fe7580aac80 con 0x7fe7580b4100 2024-03-29T11:22:25.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.254+0000 7fe755ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.254+0000 7fe755ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.254+0000 7fe755ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.254+0000 7fe755ffb640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.254+0000 7fe755ffb640 1 -- 172.21.15.59:0/3985114079 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe740003520 con 0x7fe7580b4100 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.255+0000 7fe755ffb640 1 -- 172.21.15.59:0/3985114079 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2686274099 0 0) 0x7fe740003520 con 0x7fe7580b4100 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.255+0000 7fe755ffb640 1 -- 172.21.15.59:0/3985114079 >> v1:172.21.15.59:6790/0 conn(0x7fe7580a9ac0 legacy=0x7fe7580adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.255+0000 7fe755ffb640 1 -- 172.21.15.59:0/3985114079 >> v1:172.21.15.59:6789/0 conn(0x7fe75813ee90 legacy=0x7fe75813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.256+0000 7fe755ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.256+0000 7fe755ffb640 1 monclient: found mon.noname-c 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 1 -- 172.21.15.59:0/3985114079 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75813e970 con 0x7fe7580b4100 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:25.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 1 -- 172.21.15.59:0/3985114079 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe74c001690 con 0x7fe7580b4100 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 10 monclient: dump: 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe75d647640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:22:25.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe75d647640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:25.237 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 1 -- 172.21.15.59:0/3985114079 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe74c001920 con 0x7fe7580b4100 2024-03-29T11:22:25.237 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe755ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:25.237 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe75d647640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:25.237 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.257+0000 7fe75d647640 4 set_mon_vals no callback set 2024-03-29T11:22:25.237 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.258+0000 7fe75d647640 10 monclient: shutdown 2024-03-29T11:22:25.237 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.258+0000 7fe75d647640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:25.237 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.258+0000 7fe75d647640 1 -- 172.21.15.59:0/3985114079 >> v1:172.21.15.172:6789/0 conn(0x7fe7580b4100 legacy=0x7fe758146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.258+0000 7fe75d647640 1 -- 172.21.15.59:0/3985114079 shutdown_connections 2024-03-29T11:22:25.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.258+0000 7fe75d647640 1 -- 172.21.15.59:0/3985114079 >> 172.21.15.59:0/3985114079 conn(0x7fe7580b35e0 msgr2=0x7fe7580b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:25.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.259+0000 7fe75d647640 1 -- 172.21.15.59:0/3985114079 shutdown_connections 2024-03-29T11:22:25.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.259+0000 7fe75d647640 1 -- 172.21.15.59:0/3985114079 wait complete. 2024-03-29T11:22:25.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.260+0000 7fe75d647640 10 monclient: build_initial_monmap 2024-03-29T11:22:25.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.260+0000 7fe75d647640 10 monclient: monmap: 2024-03-29T11:22:25.238 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:25.238 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:25.238 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:22:25.238 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:22:25.238 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:25.239 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:25.239 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:25.239 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:22:25.239 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:22:25.239 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:25.239 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.260+0000 7fe75d647640 1 librados: starting msgr at 2024-03-29T11:22:25.239 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.260+0000 7fe75d647640 1 librados: starting objecter 2024-03-29T11:22:25.239 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.261+0000 7fe75d647640 1 Processor -- start 2024-03-29T11:22:25.239 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.262+0000 7fe75d647640 1 -- start start 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.262+0000 7fe75d647640 1 librados: setting wanted keys 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.262+0000 7fe75d647640 1 librados: calling monclient init 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.262+0000 7fe75d647640 10 monclient: init 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.263+0000 7fe75d647640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.263+0000 7fe75d647640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.263+0000 7fe75d647640 10 monclient(hunting): picked mon.noname-c con 0x7fe7580a9ac0 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.263+0000 7fe75d647640 10 monclient(hunting): picked mon.noname-b con 0x7fe7580b4100 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.263+0000 7fe75d647640 10 monclient(hunting): picked mon.noname-a con 0x7fe75813ee90 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.263+0000 7fe75d647640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe758070720 con 0x7fe75813ee90 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.263+0000 7fe75d647640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe758070990 con 0x7fe7580b4100 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.263+0000 7fe75d647640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe758070c70 con 0x7fe7580a9ac0 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.263+0000 7fe75d647640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:25.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.263+0000 7fe75d647640 10 monclient(hunting): authenticate will time out at 995.376940s 2024-03-29T11:22:25.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.266+0000 7fe7577fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe75813ee90 0x7fe7581e6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41885/0 (socket says 172.21.15.59:41885) 2024-03-29T11:22:25.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.266+0000 7fe7577fe640 1 -- 172.21.15.59:0/3391938046 learned_addr learned my addr 172.21.15.59:0/3391938046 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:25.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 client.?.objecter ms_handle_connect 0x7fe75813ee90 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 client.?.objecter ms_handle_connect 0x7fe7580b4100 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053372218 0 0) 0x7fe758070720 con 0x7fe75813ee90 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe744004eb0 con 0x7fe75813ee90 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886104089 0 0) 0x7fe758070990 con 0x7fe7580b4100 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:22:25.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.267+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe758070720 con 0x7fe7580b4100 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1779410821 0 0) 0x7fe758070720 con 0x7fe7580b4100 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe758070990 con 0x7fe7580b4100 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2529589530 0 0) 0x7fe744004eb0 con 0x7fe75813ee90 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe758070720 con 0x7fe75813ee90 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe74c001020 con 0x7fe7580b4100 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:25.251 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe748001a50 con 0x7fe75813ee90 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.269+0000 7fe73f7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:25.252 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:25.253 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:25.253 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:25.253 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.253 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:25.253 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:25.253 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:25.253 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:25.253 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:25.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2869722859 0 0) 0x7fe758070720 con 0x7fe75813ee90 2024-03-29T11:22:25.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 >> v1:172.21.15.59:6790/0 conn(0x7fe7580a9ac0 legacy=0x7fe758071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 >> v1:172.21.15.172:6789/0 conn(0x7fe7580b4100 legacy=0x7fe758072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe73f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:25.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe73f7fe640 1 monclient: found mon.a 2024-03-29T11:22:25.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe73f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:25.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe73f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:25.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe758070f50 con 0x7fe75813ee90 2024-03-29T11:22:25.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe73f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:25.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe73f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:25.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe73f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:25.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe75d647640 5 monclient: authenticate success, global_id 4347 2024-03-29T11:22:25.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe75d647640 10 monclient: _renew_subs 2024-03-29T11:22:25.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe75d647640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:25.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe75d647640 1 -- 172.21.15.59:0/3391938046 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe758071510 con 0x7fe75813ee90 2024-03-29T11:22:25.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe75d647640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:25.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe75d647640 10 monclient: _renew_subs 2024-03-29T11:22:25.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe75d647640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:25.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe75d647640 1 -- 172.21.15.59:0/3391938046 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75806f5b0 con 0x7fe75813ee90 2024-03-29T11:22:25.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.270+0000 7fe75d647640 1 librados: init done 2024-03-29T11:22:25.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.271+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe748001bb0 con 0x7fe75813ee90 2024-03-29T11:22:25.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.271+0000 7fe73f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:25.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.271+0000 7fe754ff9640 4 set_mon_vals no callback set 2024-03-29T11:22:25.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.272+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe748004be0 con 0x7fe75813ee90 2024-03-29T11:22:25.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.272+0000 7fe73f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.272+0000 7fe73f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.272+0000 7fe73f7fe640 10 monclient: dump: 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:25.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.273+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fe748004d40 con 0x7fe75813ee90 2024-03-29T11:22:25.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.273+0000 7fe75d647640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.273+0000 7fe75d647640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.273+0000 7fe75d647640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:25.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.273+0000 7fe75d647640 1 -- 172.21.15.59:0/3391938046 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe738005080 con 0x7fe75813ee90 2024-03-29T11:22:25.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.274+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fe748084be0 con 0x7fe75813ee90 2024-03-29T11:22:25.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.274+0000 7fe73f7fe640 10 client.4347.objecter ms_dispatch 0x7fe758000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:25.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.274+0000 7fe73f7fe640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:25.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.274+0000 7fe73f7fe640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:25.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.274+0000 7fe73f7fe640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T11:22:25.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.274+0000 7fe756ffd640 10 monclient: get_auth_request con 0x7fe74406df70 auth_method 0 2024-03-29T11:22:25.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.275+0000 7fe73f7fe640 10 client.4347.objecter ms_handle_connect 0x7fe74406df70 2024-03-29T11:22:25.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.282+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7480504c0 con 0x7fe75813ee90 2024-03-29T11:22:25.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.282+0000 7fe73f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.282+0000 7fe73f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:25.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.618+0000 7fe75d647640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:25.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.618+0000 7fe75d647640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:25.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.618+0000 7fe75d647640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:25.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.618+0000 7fe75d647640 1 -- 172.21.15.59:0/3391938046 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe738005330 con 0x7fe75813ee90 2024-03-29T11:22:25.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.619+0000 7fe73f7fe640 1 -- 172.21.15.59:0/3391938046 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fe7480523f0 con 0x7fe75813ee90 2024-03-29T11:22:25.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.619+0000 7fe73f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:25.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.619+0000 7fe73f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:25.597 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:22:25.597 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:25.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.623+0000 7fe75d647640 10 librados: watch_flush enter 2024-03-29T11:22:25.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.623+0000 7fe75d647640 10 librados: watch_flush exit 2024-03-29T11:22:25.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.623+0000 7fe75d647640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T11:22:25.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.623+0000 7fe75d647640 10 client.4347.objecter successfully canceled tick 2024-03-29T11:22:25.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.624+0000 7fe75d647640 1 -- 172.21.15.59:0/3391938046 >> v1:172.21.15.172:6816/2420742773 conn(0x7fe74406df70 legacy=0x7fe744070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.624+0000 7fe75d647640 10 monclient: shutdown 2024-03-29T11:22:25.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.624+0000 7fe75d647640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:25.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.624+0000 7fe75d647640 1 -- 172.21.15.59:0/3391938046 >> v1:172.21.15.59:6789/0 conn(0x7fe75813ee90 legacy=0x7fe7581e6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.624+0000 7fe75d647640 1 -- 172.21.15.59:0/3391938046 shutdown_connections 2024-03-29T11:22:25.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.624+0000 7fe75d647640 1 -- 172.21.15.59:0/3391938046 >> 172.21.15.59:0/3391938046 conn(0x7fe7580b35e0 msgr2=0x7fe7580b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:25.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.624+0000 7fe75d647640 1 -- 172.21.15.59:0/3391938046 shutdown_connections 2024-03-29T11:22:25.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.624+0000 7fe75d647640 1 -- 172.21.15.59:0/3391938046 wait complete. 2024-03-29T11:22:25.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:25.625+0000 7fe75d647640 1 librados: shutdown 2024-03-29T11:22:25.619 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:32.426 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:32.590 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.612+0000 7f43467c4640 10 monclient: get_monmap_and_config 2024-03-29T11:22:32.590 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.612+0000 7f43467c4640 10 monclient: build_initial_monmap 2024-03-29T11:22:32.590 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.612+0000 7f43467c4640 10 monclient: monmap: 2024-03-29T11:22:32.590 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:32.590 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:32.590 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:22:32.613365+0000 2024-03-29T11:22:32.590 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:22:32.613365+0000 2024-03-29T11:22:32.590 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:32.590 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:32.590 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:32.590 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:22:32.591 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:22:32.591 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:32.591 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.614+0000 7f43467c4640 1 Processor -- start 2024-03-29T11:22:32.591 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.614+0000 7f43467c4640 1 -- start start 2024-03-29T11:22:32.591 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.614+0000 7f43467c4640 10 monclient: init 2024-03-29T11:22:32.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.615+0000 7f43467c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:32.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.615+0000 7f43467c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:32.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.619+0000 7f43467c4640 10 monclient(hunting): picked mon.noname-b con 0x7f43400b4100 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:32.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.619+0000 7f43467c4640 10 monclient(hunting): picked mon.noname-c con 0x7f434013ee90 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:32.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.619+0000 7f43467c4640 10 monclient(hunting): picked mon.noname-a con 0x7f43400af900 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:32.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.619+0000 7f43467c4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400afd10 con 0x7f43400af900 2024-03-29T11:22:32.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.619+0000 7f43467c4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400af6f0 con 0x7f434013ee90 2024-03-29T11:22:32.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.619+0000 7f43467c4640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400aac80 con 0x7f43400b4100 2024-03-29T11:22:32.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.619+0000 7f43467c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:32.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.619+0000 7f43467c4640 10 monclient(hunting): authenticate will time out at 1002.732803s 2024-03-29T11:22:32.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.619+0000 7f433ffff640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f43400b4100 0x7f43400adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:41623/0 (socket says 172.21.15.59:41623) 2024-03-29T11:22:32.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.619+0000 7f433ffff640 1 -- 172.21.15.59:0/3628183794 learned_addr learned my addr 172.21.15.59:0/3628183794 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:32.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 1 -- 172.21.15.59:0/3628183794 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400681082 0 0) 0x7f43400aac80 con 0x7f43400b4100 2024-03-29T11:22:32.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:32.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:32.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:32.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:22:32.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 1 -- 172.21.15.59:0/3628183794 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4328003520 con 0x7f43400b4100 2024-03-29T11:22:32.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 1 -- 172.21.15.59:0/3628183794 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1510977618 0 0) 0x7f4328003520 con 0x7f43400b4100 2024-03-29T11:22:32.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 1 -- 172.21.15.59:0/3628183794 >> v1:172.21.15.172:6789/0 conn(0x7f434013ee90 legacy=0x7f434013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:32.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 1 -- 172.21.15.59:0/3628183794 >> v1:172.21.15.59:6789/0 conn(0x7f43400af900 legacy=0x7f434014ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:32.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:32.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 1 monclient: found mon.noname-b 2024-03-29T11:22:32.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:32.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:22:32.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 1 -- 172.21.15.59:0/3628183794 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400aaf60 con 0x7f43400b4100 2024-03-29T11:22:32.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 10 monclient: _finish_auth 0 2024-03-29T11:22:32.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:32.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:32.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 1 -- 172.21.15.59:0/3628183794 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4338001a10 con 0x7f43400b4100 2024-03-29T11:22:32.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:32.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:32.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.622+0000 7f433effd640 10 monclient: dump: 2024-03-29T11:22:32.605 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:32.605 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:32.605 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:32.605 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:32.605 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:32.606 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:32.606 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:32.606 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:32.606 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:32.606 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:32.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.623+0000 7f43467c4640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:22:32.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.623+0000 7f43467c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:32.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.623+0000 7f433effd640 1 -- 172.21.15.59:0/3628183794 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4338001be0 con 0x7f43400b4100 2024-03-29T11:22:32.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.623+0000 7f433effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:32.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.623+0000 7f433effd640 1 -- 172.21.15.59:0/3628183794 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4338002030 con 0x7f43400b4100 2024-03-29T11:22:32.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.623+0000 7f433effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:32.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.623+0000 7f433effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:32.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.623+0000 7f433effd640 10 monclient: dump: 2024-03-29T11:22:32.608 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:32.608 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:32.608 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:32.608 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:32.608 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:32.609 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:32.609 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:32.609 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:32.609 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:32.609 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:32.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.623+0000 7f43467c4640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:32.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.623+0000 7f43467c4640 4 set_mon_vals no callback set 2024-03-29T11:22:32.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.624+0000 7f43467c4640 10 monclient: shutdown 2024-03-29T11:22:32.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.624+0000 7f43467c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:32.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.624+0000 7f43467c4640 1 -- 172.21.15.59:0/3628183794 >> v1:172.21.15.59:6790/0 conn(0x7f43400b4100 legacy=0x7f43400adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:32.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.624+0000 7f43467c4640 1 -- 172.21.15.59:0/3628183794 shutdown_connections 2024-03-29T11:22:32.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.624+0000 7f43467c4640 1 -- 172.21.15.59:0/3628183794 >> 172.21.15.59:0/3628183794 conn(0x7f43400b35e0 msgr2=0x7f43400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:32.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.626+0000 7f43467c4640 1 -- 172.21.15.59:0/3628183794 shutdown_connections 2024-03-29T11:22:32.617 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.627+0000 7f43467c4640 1 -- 172.21.15.59:0/3628183794 wait complete. 2024-03-29T11:22:32.617 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.627+0000 7f43467c4640 10 monclient: build_initial_monmap 2024-03-29T11:22:32.617 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.627+0000 7f43467c4640 10 monclient: monmap: 2024-03-29T11:22:32.617 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.628+0000 7f43467c4640 1 librados: starting msgr at 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.628+0000 7f43467c4640 1 librados: starting objecter 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.628+0000 7f43467c4640 1 Processor -- start 2024-03-29T11:22:32.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.628+0000 7f43467c4640 1 -- start start 2024-03-29T11:22:32.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.628+0000 7f43467c4640 1 librados: setting wanted keys 2024-03-29T11:22:32.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.628+0000 7f43467c4640 1 librados: calling monclient init 2024-03-29T11:22:32.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.628+0000 7f43467c4640 10 monclient: init 2024-03-29T11:22:32.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.629+0000 7f43467c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:32.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.629+0000 7f43467c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:32.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.632+0000 7f433ffff640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f43400af900 0x7f43401d3410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:41721/0 (socket says 172.21.15.59:41721) 2024-03-29T11:22:32.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.632+0000 7f433ffff640 1 -- 172.21.15.59:0/245964473 learned_addr learned my addr 172.21.15.59:0/245964473 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:32.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.632+0000 7f43467c4640 10 monclient(hunting): picked mon.noname-c con 0x7f43400af900 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:32.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.632+0000 7f43467c4640 10 monclient(hunting): picked mon.noname-a con 0x7f43400b4100 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:32.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.632+0000 7f43467c4640 10 monclient(hunting): picked mon.noname-b con 0x7f434013ee90 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:32.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.632+0000 7f43467c4640 1 -- 172.21.15.59:0/245964473 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43401423b0 con 0x7f43400b4100 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.632+0000 7f43467c4640 1 -- 172.21.15.59:0/245964473 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43401d2ef0 con 0x7f434013ee90 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.632+0000 7f43467c4640 1 -- 172.21.15.59:0/245964473 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43401d3b20 con 0x7f43400af900 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.632+0000 7f43467c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.632+0000 7f43467c4640 10 monclient(hunting): authenticate will time out at 1002.746505s 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.632+0000 7f433cff9640 10 client.?.objecter ms_handle_connect 0x7f43400af900 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.633+0000 7f433cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.633+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264672856 0 0) 0x7f43401d3b20 con 0x7f43400af900 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.633+0000 7f433cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.633+0000 7f433cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.633+0000 7f433cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.633+0000 7f433cff9640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.633+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f432c004eb0 con 0x7f43400af900 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.634+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2107361864 0 0) 0x7f432c004eb0 con 0x7f43400af900 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.634+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43401d3b20 con 0x7f43400af900 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.634+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4338001ef0 con 0x7f43400af900 2024-03-29T11:22:32.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.634+0000 7f433cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.634+0000 7f433cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.634+0000 7f433cff9640 10 monclient(hunting): dump: 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.634+0000 7f433cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.634+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3761794354 0 0) 0x7f43401d3b20 con 0x7f43400af900 2024-03-29T11:22:32.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.634+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 >> v1:172.21.15.172:6789/0 conn(0x7f434013ee90 legacy=0x7f4340140c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.635+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 >> v1:172.21.15.59:6789/0 conn(0x7f43400b4100 legacy=0x7f43401367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 1 monclient: found mon.c 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4340136ed0 con 0x7f43400af900 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f43467c4640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f43467c4640 10 monclient: _renew_subs 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f43467c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f43467c4640 1 -- 172.21.15.59:0/245964473 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4340137140 con 0x7f43400af900 2024-03-29T11:22:32.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f43467c4640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f43467c4640 10 monclient: _renew_subs 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4338004f70 con 0x7f43400af900 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f43467c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f43467c4640 1 -- 172.21.15.59:0/245964473 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f434013cc00 con 0x7f43400af900 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f43467c4640 1 librados: init done 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f43380053c0 con 0x7f43400af900 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433cff9640 10 monclient: dump: 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:32.623 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.636+0000 7f433e7fc640 4 set_mon_vals no callback set 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.638+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f4338005520 con 0x7f43400af900 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.639+0000 7f43267fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.639+0000 7f43267fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.639+0000 7f43267fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.639+0000 7f43267fc640 1 -- 172.21.15.59:0/245964473 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4318005080 con 0x7f43400af900 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.639+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f4338085060 con 0x7f43400af900 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.639+0000 7f433cff9640 10 client.4325.objecter ms_dispatch 0x7f4340000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:32.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.639+0000 7f433cff9640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:32.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.639+0000 7f433cff9640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:32.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.640+0000 7f433cff9640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T11:22:32.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.643+0000 7f433f7fe640 10 monclient: get_auth_request con 0x7f432c06dd20 auth_method 0 2024-03-29T11:22:32.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.643+0000 7f433cff9640 10 client.4325.objecter ms_handle_connect 0x7f432c06dd20 2024-03-29T11:22:32.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.649+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4338050940 con 0x7f43400af900 2024-03-29T11:22:32.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.649+0000 7f433cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:32.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.649+0000 7f433cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:32.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.984+0000 7f43267fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:32.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.984+0000 7f43267fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:32.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.984+0000 7f43267fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:22:32.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.984+0000 7f43267fc640 1 -- 172.21.15.59:0/245964473 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4318005330 con 0x7f43400af900 2024-03-29T11:22:32.962 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.985+0000 7f433cff9640 1 -- 172.21.15.59:0/245964473 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f4338052870 con 0x7f43400af900 2024-03-29T11:22:32.962 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.986+0000 7f433cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:32.962 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.986+0000 7f433cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:32.962 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:22:32.962 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:32.966 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.989+0000 7f43267fc640 10 librados: watch_flush enter 2024-03-29T11:22:32.966 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.989+0000 7f43267fc640 10 librados: watch_flush exit 2024-03-29T11:22:32.966 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.989+0000 7f43267fc640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T11:22:32.966 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.989+0000 7f43267fc640 10 client.4325.objecter successfully canceled tick 2024-03-29T11:22:32.966 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.989+0000 7f43267fc640 1 -- 172.21.15.59:0/245964473 >> v1:172.21.15.172:6816/2420742773 conn(0x7f432c06dd20 legacy=0x7f432c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:32.966 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.989+0000 7f43267fc640 10 monclient: shutdown 2024-03-29T11:22:32.966 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.989+0000 7f43267fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:32.967 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.989+0000 7f43267fc640 1 -- 172.21.15.59:0/245964473 >> v1:172.21.15.59:6790/0 conn(0x7f43400af900 legacy=0x7f43401d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:32.967 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.989+0000 7f43267fc640 1 -- 172.21.15.59:0/245964473 shutdown_connections 2024-03-29T11:22:32.967 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.989+0000 7f43267fc640 1 -- 172.21.15.59:0/245964473 >> 172.21.15.59:0/245964473 conn(0x7f43400b35e0 msgr2=0x7f43400b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:32.972 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.996+0000 7f43267fc640 1 -- 172.21.15.59:0/245964473 shutdown_connections 2024-03-29T11:22:32.972 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.996+0000 7f43267fc640 1 -- 172.21.15.59:0/245964473 wait complete. 2024-03-29T11:22:32.972 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:32.996+0000 7f43267fc640 1 librados: shutdown 2024-03-29T11:22:32.990 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:39.796 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:39.944 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.967+0000 7f9c28c24640 10 monclient: get_monmap_and_config 2024-03-29T11:22:39.944 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.967+0000 7f9c28c24640 10 monclient: build_initial_monmap 2024-03-29T11:22:39.944 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.967+0000 7f9c28c24640 10 monclient: monmap: 2024-03-29T11:22:39.944 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:39.944 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:39.944 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:22:39.968123+0000 2024-03-29T11:22:39.944 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:22:39.968123+0000 2024-03-29T11:22:39.944 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:39.944 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:39.944 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:39.945 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:22:39.945 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:22:39.945 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.969+0000 7f9c28c24640 1 Processor -- start 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.969+0000 7f9c28c24640 1 -- start start 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.969+0000 7f9c28c24640 10 monclient: init 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.970+0000 7f9c28c24640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.970+0000 7f9c28c24640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.970+0000 7f9c28c24640 10 monclient(hunting): picked mon.noname-a con 0x7f9c1c0389e0 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.970+0000 7f9c28c24640 10 monclient(hunting): picked mon.noname-b con 0x7f9c1c038dc0 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.970+0000 7f9c28c24640 10 monclient(hunting): picked mon.noname-c con 0x7f9c1c030660 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.970+0000 7f9c28c24640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c0c54c0 con 0x7f9c1c0389e0 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.970+0000 7f9c28c24640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c036430 con 0x7f9c1c030660 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.970+0000 7f9c28c24640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c039840 con 0x7f9c1c038dc0 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.970+0000 7f9c28c24640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:39.947 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.970+0000 7f9c28c24640 10 monclient(hunting): authenticate will time out at 1010.084148s 2024-03-29T11:22:39.950 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.974+0000 7f9c22ffd640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f9c1c038dc0 0x7f9c1c0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:56467/0 (socket says 172.21.15.59:56467) 2024-03-29T11:22:39.950 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.974+0000 7f9c22ffd640 1 -- 172.21.15.59:0/313881555 learned_addr learned my addr 172.21.15.59:0/313881555 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:39.954 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 1 -- 172.21.15.59:0/313881555 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506363400 0 0) 0x7f9c1c039840 con 0x7f9c1c038dc0 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 1 -- 172.21.15.59:0/313881555 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c14003520 con 0x7f9c1c038dc0 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 1 -- 172.21.15.59:0/313881555 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2877613284 0 0) 0x7f9c14003520 con 0x7f9c1c038dc0 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 1 -- 172.21.15.59:0/313881555 >> v1:172.21.15.172:6789/0 conn(0x7f9c1c030660 legacy=0x7f9c1c0d3170 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 1 -- 172.21.15.59:0/313881555 >> v1:172.21.15.59:6789/0 conn(0x7f9c1c0389e0 legacy=0x7f9c1c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 1 monclient: found mon.noname-b 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 1 -- 172.21.15.59:0/313881555 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c035690 con 0x7f9c1c038dc0 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:39.955 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 1 -- 172.21.15.59:0/313881555 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9c2404f730 con 0x7f9c1c038dc0 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.975+0000 7f9c227fc640 10 monclient: dump: 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:39.956 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.976+0000 7f9c227fc640 1 -- 172.21.15.59:0/313881555 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c240008d0 con 0x7f9c1c038dc0 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.976+0000 7f9c227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.976+0000 7f9c227fc640 1 -- 172.21.15.59:0/313881555 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9c2404f730 con 0x7f9c1c038dc0 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.976+0000 7f9c227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.976+0000 7f9c227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.976+0000 7f9c227fc640 10 monclient: dump: 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:39.957 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:39.958 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:39.958 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:39.958 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:39.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.977+0000 7f9c28c24640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:22:39.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.977+0000 7f9c28c24640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:39.958 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.977+0000 7f9c28c24640 4 set_mon_vals no callback set 2024-03-29T11:22:39.959 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.978+0000 7f9c28c24640 10 monclient: shutdown 2024-03-29T11:22:39.959 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.978+0000 7f9c28c24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:39.959 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.978+0000 7f9c28c24640 1 -- 172.21.15.59:0/313881555 >> v1:172.21.15.59:6790/0 conn(0x7f9c1c038dc0 legacy=0x7f9c1c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:39.959 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.979+0000 7f9c28c24640 1 -- 172.21.15.59:0/313881555 shutdown_connections 2024-03-29T11:22:39.959 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.979+0000 7f9c28c24640 1 -- 172.21.15.59:0/313881555 >> 172.21.15.59:0/313881555 conn(0x7f9c1c005c00 msgr2=0x7f9c1c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:39.960 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.979+0000 7f9c28c24640 1 -- 172.21.15.59:0/313881555 shutdown_connections 2024-03-29T11:22:39.960 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.980+0000 7f9c28c24640 1 -- 172.21.15.59:0/313881555 wait complete. 2024-03-29T11:22:39.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.981+0000 7f9c28c24640 10 monclient: build_initial_monmap 2024-03-29T11:22:39.961 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.981+0000 7f9c28c24640 10 monclient: monmap: 2024-03-29T11:22:39.961 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:39.961 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:39.961 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:22:39.961 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.981+0000 7f9c28c24640 1 librados: starting msgr at 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.981+0000 7f9c28c24640 1 librados: starting objecter 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.981+0000 7f9c28c24640 1 Processor -- start 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.982+0000 7f9c28c24640 1 -- start start 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.982+0000 7f9c28c24640 1 librados: setting wanted keys 2024-03-29T11:22:39.962 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.982+0000 7f9c28c24640 1 librados: calling monclient init 2024-03-29T11:22:39.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.982+0000 7f9c28c24640 10 monclient: init 2024-03-29T11:22:39.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.983+0000 7f9c28c24640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:39.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.983+0000 7f9c28c24640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:39.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.983+0000 7f9c28c24640 10 monclient(hunting): picked mon.noname-c con 0x7f9c1c030660 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:39.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.983+0000 7f9c28c24640 10 monclient(hunting): picked mon.noname-b con 0x7f9c1c0389e0 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:39.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.983+0000 7f9c28c24640 10 monclient(hunting): picked mon.noname-a con 0x7f9c1c15fe90 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:39.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.983+0000 7f9c28c24640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c0c0ee0 con 0x7f9c1c15fe90 2024-03-29T11:22:39.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.983+0000 7f9c28c24640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c0c73e0 con 0x7f9c1c0389e0 2024-03-29T11:22:39.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.983+0000 7f9c28c24640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c0c8590 con 0x7f9c1c030660 2024-03-29T11:22:39.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.983+0000 7f9c28c24640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:39.963 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.983+0000 7f9c28c24640 10 monclient(hunting): authenticate will time out at 1010.097100s 2024-03-29T11:22:39.964 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.987+0000 7f9c22ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9c1c0389e0 0x7f9c1c02fad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:43207/0 (socket says 172.21.15.59:43207) 2024-03-29T11:22:39.964 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.987+0000 7f9c22ffd640 1 -- 172.21.15.59:0/3643002318 learned_addr learned my addr 172.21.15.59:0/3643002318 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:39.967 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.989+0000 7f9c0ffff640 10 client.?.objecter ms_handle_connect 0x7f9c1c0389e0 2024-03-29T11:22:39.967 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.989+0000 7f9c0ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:39.967 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.989+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918716839 0 0) 0x7f9c1c0c73e0 con 0x7f9c1c0389e0 2024-03-29T11:22:39.967 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.989+0000 7f9c0ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:39.967 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.989+0000 7f9c0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:39.967 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.989+0000 7f9c0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:39.968 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.989+0000 7f9c0ffff640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:22:39.968 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.989+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c08004eb0 con 0x7f9c1c0389e0 2024-03-29T11:22:39.968 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.990+0000 7f9c0ffff640 10 client.?.objecter ms_handle_connect 0x7f9c1c15fe90 2024-03-29T11:22:39.968 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.990+0000 7f9c0ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:39.968 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.990+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255921556 0 0) 0x7f9c18001090 con 0x7f9c1c15fe90 2024-03-29T11:22:39.968 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.990+0000 7f9c0ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:39.968 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.990+0000 7f9c0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:39.968 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.990+0000 7f9c0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:39.968 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.990+0000 7f9c0ffff640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:22:39.968 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.990+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c1c0c73e0 con 0x7f9c1c15fe90 2024-03-29T11:22:39.969 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.992+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1626648783 0 0) 0x7f9c1c0c73e0 con 0x7f9c1c15fe90 2024-03-29T11:22:39.969 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.992+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c18001090 con 0x7f9c1c15fe90 2024-03-29T11:22:39.969 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.992+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9c180017e0 con 0x7f9c1c15fe90 2024-03-29T11:22:39.969 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.992+0000 7f9c0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:39.969 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.992+0000 7f9c0ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:39.969 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.992+0000 7f9c0ffff640 10 monclient(hunting): dump: 2024-03-29T11:22:39.969 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.992+0000 7f9c0ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3317641123 0 0) 0x7f9c08004eb0 con 0x7f9c1c0389e0 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c1c0c73e0 con 0x7f9c1c0389e0 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9c2404f730 con 0x7f9c1c0389e0 2024-03-29T11:22:39.970 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 10 monclient(hunting): dump: 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:39.971 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:39.972 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 564017230 0 0) 0x7f9c18001090 con 0x7f9c1c15fe90 2024-03-29T11:22:39.972 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 >> v1:172.21.15.59:6790/0 conn(0x7f9c1c030660 legacy=0x7f9c1c0c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:39.972 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 >> v1:172.21.15.172:6789/0 conn(0x7f9c1c0389e0 legacy=0x7f9c1c02fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:39.972 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:39.972 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 1 monclient: found mon.a 2024-03-29T11:22:39.972 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:39.972 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:39.972 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c0c8720 con 0x7f9c1c15fe90 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 10 monclient: _finish_auth 0 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c28c24640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c28c24640 10 monclient: _renew_subs 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c28c24640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c28c24640 1 -- 172.21.15.59:0/3643002318 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1c160510 con 0x7f9c1c15fe90 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c28c24640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c28c24640 10 monclient: _renew_subs 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c28c24640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c28c24640 1 -- 172.21.15.59:0/3643002318 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1c1609f0 con 0x7f9c1c15fe90 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.993+0000 7f9c28c24640 1 librados: init done 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.994+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c18001db0 con 0x7f9c1c15fe90 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.994+0000 7f9c0ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:39.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.994+0000 7f9c21ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.994+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9c18003830 con 0x7f9c1c15fe90 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.994+0000 7f9c0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.994+0000 7f9c0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.994+0000 7f9c0ffff640 10 monclient: dump: 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:39.974 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:39.975 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.997+0000 7f9c28c24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:39.975 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.997+0000 7f9c28c24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:39.975 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.997+0000 7f9c28c24640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:39.975 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.997+0000 7f9c28c24640 1 -- 172.21.15.59:0/3643002318 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c04005080 con 0x7f9c1c15fe90 2024-03-29T11:22:39.975 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:39.999+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f9c18019880 con 0x7f9c1c15fe90 2024-03-29T11:22:39.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.000+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f9c18083820 con 0x7f9c1c15fe90 2024-03-29T11:22:39.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.000+0000 7f9c0ffff640 10 client.4353.objecter ms_dispatch 0x7f9c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:39.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.000+0000 7f9c0ffff640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:39.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.000+0000 7f9c0ffff640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:39.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.000+0000 7f9c0ffff640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T11:22:39.981 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.005+0000 7f9c237fe640 10 monclient: get_auth_request con 0x7f9c0806dcf0 auth_method 0 2024-03-29T11:22:39.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.005+0000 7f9c0ffff640 10 client.4353.objecter ms_handle_connect 0x7f9c0806dcf0 2024-03-29T11:22:39.987 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.010+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c1804ef30 con 0x7f9c1c15fe90 2024-03-29T11:22:39.987 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.010+0000 7f9c0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:39.987 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.010+0000 7f9c0ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:40.309 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.332+0000 7f9c28c24640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.309 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.332+0000 7f9c28c24640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.309 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.332+0000 7f9c28c24640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:22:40.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.332+0000 7f9c28c24640 1 -- 172.21.15.59:0/3643002318 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c04005330 con 0x7f9c1c15fe90 2024-03-29T11:22:40.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.336+0000 7f9c0ffff640 1 -- 172.21.15.59:0/3643002318 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f9c18051030 con 0x7f9c1c15fe90 2024-03-29T11:22:40.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.336+0000 7f9c0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.336+0000 7f9c0ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.340+0000 7f9c28c24640 10 librados: watch_flush enter 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.340+0000 7f9c28c24640 10 librados: watch_flush exit 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.340+0000 7f9c28c24640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.340+0000 7f9c28c24640 10 client.4353.objecter successfully canceled tick 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.340+0000 7f9c28c24640 1 -- 172.21.15.59:0/3643002318 >> v1:172.21.15.172:6816/2420742773 conn(0x7f9c0806dcf0 legacy=0x7f9c080701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.341+0000 7f9c28c24640 10 monclient: shutdown 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.341+0000 7f9c28c24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.341+0000 7f9c28c24640 1 -- 172.21.15.59:0/3643002318 >> v1:172.21.15.59:6789/0 conn(0x7f9c1c15fe90 legacy=0x7f9c1c0c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.341+0000 7f9c28c24640 1 -- 172.21.15.59:0/3643002318 shutdown_connections 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.341+0000 7f9c28c24640 1 -- 172.21.15.59:0/3643002318 >> 172.21.15.59:0/3643002318 conn(0x7f9c1c005c00 msgr2=0x7f9c1c034b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:40.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.341+0000 7f9c28c24640 1 -- 172.21.15.59:0/3643002318 shutdown_connections 2024-03-29T11:22:40.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.341+0000 7f9c28c24640 1 -- 172.21.15.59:0/3643002318 wait complete. 2024-03-29T11:22:40.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:40.341+0000 7f9c28c24640 1 librados: shutdown 2024-03-29T11:22:40.334 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:47.139 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:47.302 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.325+0000 7fb08f34b640 10 monclient: get_monmap_and_config 2024-03-29T11:22:47.302 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.325+0000 7fb08f34b640 10 monclient: build_initial_monmap 2024-03-29T11:22:47.302 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.325+0000 7fb08f34b640 10 monclient: monmap: 2024-03-29T11:22:47.302 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:47.302 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:47.302 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:22:47.325914+0000 2024-03-29T11:22:47.302 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:22:47.325914+0000 2024-03-29T11:22:47.302 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:47.302 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:47.303 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:47.303 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:22:47.303 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:22:47.303 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:47.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.326+0000 7fb08f34b640 1 Processor -- start 2024-03-29T11:22:47.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.326+0000 7fb08f34b640 1 -- start start 2024-03-29T11:22:47.303 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.326+0000 7fb08f34b640 10 monclient: init 2024-03-29T11:22:47.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.327+0000 7fb08f34b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:47.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.327+0000 7fb08f34b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:47.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.327+0000 7fb08f34b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0880b0770 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:47.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.327+0000 7fb08f34b640 10 monclient(hunting): picked mon.noname-c con 0x7fb08814a700 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:47.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.330+0000 7fb08f34b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0880ac5a0 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:47.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.330+0000 7fb08f34b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0880aba70 con 0x7fb0880b0770 2024-03-29T11:22:47.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.330+0000 7fb08f34b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0880aecb0 con 0x7fb08814a700 2024-03-29T11:22:47.308 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.330+0000 7fb08f34b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0880af150 con 0x7fb0880ac5a0 2024-03-29T11:22:47.308 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.330+0000 7fb08f34b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:47.308 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.330+0000 7fb08f34b640 10 monclient(hunting): authenticate will time out at 1017.444678s 2024-03-29T11:22:47.308 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.331+0000 7fb08d8e8640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fb0880ac5a0 0x7fb088151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:44897/0 (socket says 172.21.15.59:44897) 2024-03-29T11:22:47.308 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.331+0000 7fb08d8e8640 1 -- 172.21.15.59:0/3020622403 learned_addr learned my addr 172.21.15.59:0/3020622403 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:47.309 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.332+0000 7fb07ffff640 1 -- 172.21.15.59:0/3020622403 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052988110 0 0) 0x7fb0880aba70 con 0x7fb0880b0770 2024-03-29T11:22:47.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.332+0000 7fb07ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.332+0000 7fb07ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.332+0000 7fb07ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:22:47.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 1 -- 172.21.15.59:0/3020622403 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb074003520 con 0x7fb0880b0770 2024-03-29T11:22:47.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 1 -- 172.21.15.59:0/3020622403 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 548614619 0 0) 0x7fb074003520 con 0x7fb0880b0770 2024-03-29T11:22:47.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 1 -- 172.21.15.59:0/3020622403 >> v1:172.21.15.59:6790/0 conn(0x7fb0880ac5a0 legacy=0x7fb088151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 1 -- 172.21.15.59:0/3020622403 >> v1:172.21.15.172:6789/0 conn(0x7fb08814a700 legacy=0x7fb0880abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:47.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:47.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 1 monclient: found mon.noname-a 2024-03-29T11:22:47.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:47.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:22:47.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 1 -- 172.21.15.59:0/3020622403 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0880ac9d0 con 0x7fb0880b0770 2024-03-29T11:22:47.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 10 monclient: _finish_auth 0 2024-03-29T11:22:47.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 1 -- 172.21.15.59:0/3020622403 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb078001690 con 0x7fb0880b0770 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb08f34b640 5 monclient: authenticate success, global_id 4356 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb08f34b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.333+0000 7fb07ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.334+0000 7fb07ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.334+0000 7fb07ffff640 10 monclient: dump: 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:47.314 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:47.315 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:47.315 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:47.315 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:47.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.334+0000 7fb07ffff640 1 -- 172.21.15.59:0/3020622403 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb078001920 con 0x7fb0880b0770 2024-03-29T11:22:47.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.334+0000 7fb08f34b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:47.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.334+0000 7fb07ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:47.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.334+0000 7fb07ffff640 1 -- 172.21.15.59:0/3020622403 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb078001cb0 con 0x7fb0880b0770 2024-03-29T11:22:47.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.334+0000 7fb08f34b640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:47.315 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.334+0000 7fb08f34b640 4 set_mon_vals no callback set 2024-03-29T11:22:47.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.335+0000 7fb08f34b640 10 monclient: shutdown 2024-03-29T11:22:47.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.335+0000 7fb07ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.335+0000 7fb07ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:47.316 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.335+0000 7fb07ffff640 10 monclient: dump: 2024-03-29T11:22:47.316 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:47.316 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:47.317 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:47.317 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:47.317 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.317 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:47.317 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:47.317 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:47.317 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:47.317 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:47.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.335+0000 7fb08f34b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:47.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.335+0000 7fb08f34b640 1 -- 172.21.15.59:0/3020622403 >> v1:172.21.15.59:6789/0 conn(0x7fb0880b0770 legacy=0x7fb088149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.336+0000 7fb08f34b640 1 -- 172.21.15.59:0/3020622403 shutdown_connections 2024-03-29T11:22:47.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.336+0000 7fb08f34b640 1 -- 172.21.15.59:0/3020622403 >> 172.21.15.59:0/3020622403 conn(0x7fb08806b660 msgr2=0x7fb08806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.338+0000 7fb08f34b640 1 -- 172.21.15.59:0/3020622403 shutdown_connections 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.338+0000 7fb08f34b640 1 -- 172.21.15.59:0/3020622403 wait complete. 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.339+0000 7fb08f34b640 10 monclient: build_initial_monmap 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.339+0000 7fb08f34b640 10 monclient: monmap: 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:22:47.320 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:47.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.340+0000 7fb08f34b640 1 librados: starting msgr at 2024-03-29T11:22:47.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.340+0000 7fb08f34b640 1 librados: starting objecter 2024-03-29T11:22:47.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.340+0000 7fb08f34b640 1 Processor -- start 2024-03-29T11:22:47.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.340+0000 7fb08f34b640 1 -- start start 2024-03-29T11:22:47.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.340+0000 7fb08f34b640 1 librados: setting wanted keys 2024-03-29T11:22:47.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.340+0000 7fb08f34b640 1 librados: calling monclient init 2024-03-29T11:22:47.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.340+0000 7fb08f34b640 10 monclient: init 2024-03-29T11:22:47.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.341+0000 7fb08f34b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:47.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.341+0000 7fb08f34b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:47.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.341+0000 7fb08f34b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0880ac5a0 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:47.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.341+0000 7fb08f34b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0880b0770 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:47.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.345+0000 7fb08f34b640 10 monclient(hunting): picked mon.noname-a con 0x7fb08814a700 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:47.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.345+0000 7fb08f34b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb088128080 con 0x7fb08814a700 2024-03-29T11:22:47.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.345+0000 7fb08f34b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb088127ca0 con 0x7fb0880ac5a0 2024-03-29T11:22:47.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.345+0000 7fb08f34b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb088127e30 con 0x7fb0880b0770 2024-03-29T11:22:47.325 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.348+0000 7fb08f34b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:47.325 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.349+0000 7fb08f34b640 10 monclient(hunting): authenticate will time out at 1017.462798s 2024-03-29T11:22:47.326 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.349+0000 7fb08d0e7640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb0880ac5a0 0x7fb088146ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:33151/0 (socket says 172.21.15.59:33151) 2024-03-29T11:22:47.326 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.349+0000 7fb08d0e7640 1 -- 172.21.15.59:0/3156534660 learned_addr learned my addr 172.21.15.59:0/3156534660 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:47.326 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.349+0000 7fb07dffb640 10 client.?.objecter ms_handle_connect 0x7fb0880ac5a0 2024-03-29T11:22:47.326 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.349+0000 7fb07dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:47.327 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.350+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854355773 0 0) 0x7fb088127ca0 con 0x7fb0880ac5a0 2024-03-29T11:22:47.327 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.350+0000 7fb07dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.327 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.350+0000 7fb07dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.327 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.350+0000 7fb07dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.327 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.350+0000 7fb07dffb640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:22:47.328 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.350+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb06c004eb0 con 0x7fb0880ac5a0 2024-03-29T11:22:47.328 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.351+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1818526230 0 0) 0x7fb06c004eb0 con 0x7fb0880ac5a0 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.351+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb088127ca0 con 0x7fb0880ac5a0 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.351+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb078001650 con 0x7fb0880ac5a0 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.351+0000 7fb07dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.352+0000 7fb07dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.352+0000 7fb07dffb640 10 monclient(hunting): dump: 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:47.329 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:47.330 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.352+0000 7fb07dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:47.330 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.353+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1372531715 0 0) 0x7fb088127ca0 con 0x7fb0880ac5a0 2024-03-29T11:22:47.330 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.353+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 >> v1:172.21.15.59:6790/0 conn(0x7fb0880b0770 legacy=0x7fb0881285a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.330 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.353+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 >> v1:172.21.15.59:6789/0 conn(0x7fb08814a700 legacy=0x7fb088128cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:47.330 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.353+0000 7fb07dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:47.330 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.353+0000 7fb07dffb640 1 monclient: found mon.b 2024-03-29T11:22:47.330 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.353+0000 7fb07dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:47.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.353+0000 7fb07dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:22:47.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.353+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0881df580 con 0x7fb0880ac5a0 2024-03-29T11:22:47.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.353+0000 7fb07dffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:47.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.353+0000 7fb07dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:47.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.353+0000 7fb07dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:47.332 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb08f34b640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:22:47.332 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb08f34b640 10 monclient: _renew_subs 2024-03-29T11:22:47.332 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb08f34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:22:47.332 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb08f34b640 1 -- 172.21.15.59:0/3156534660 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0881df7d0 con 0x7fb0880ac5a0 2024-03-29T11:22:47.332 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb078003960 con 0x7fb0880ac5a0 2024-03-29T11:22:47.332 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb07dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb078003d30 con 0x7fb0880ac5a0 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb07dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb07f7fe640 4 set_mon_vals no callback set 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb08f34b640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb07dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.354+0000 7fb07dffb640 10 monclient: dump: 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:47.333 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:47.334 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:47.334 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:47.334 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.355+0000 7fb08f34b640 10 monclient: _renew_subs 2024-03-29T11:22:47.334 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.355+0000 7fb08f34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:22:47.334 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.355+0000 7fb08f34b640 1 -- 172.21.15.59:0/3156534660 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb088147d50 con 0x7fb0880ac5a0 2024-03-29T11:22:47.334 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.355+0000 7fb08f34b640 1 librados: init done 2024-03-29T11:22:47.334 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.357+0000 7fb08f34b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.334 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.357+0000 7fb08f34b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.357+0000 7fb08f34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:22:47.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.357+0000 7fb08f34b640 1 -- 172.21.15.59:0/3156534660 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0881282c0 con 0x7fb0880ac5a0 2024-03-29T11:22:47.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.357+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fb078003e90 con 0x7fb0880ac5a0 2024-03-29T11:22:47.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.358+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fb078084c90 con 0x7fb0880ac5a0 2024-03-29T11:22:47.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.358+0000 7fb07dffb640 10 client.4330.objecter ms_dispatch 0x7fb088000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:47.336 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.358+0000 7fb07dffb640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:47.336 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.358+0000 7fb07dffb640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:47.336 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.358+0000 7fb07dffb640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T11:22:47.337 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.360+0000 7fb08c8e6640 10 monclient: get_auth_request con 0x7fb06c06dc90 auth_method 0 2024-03-29T11:22:47.337 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.361+0000 7fb07dffb640 10 client.4330.objecter ms_handle_connect 0x7fb06c06dc90 2024-03-29T11:22:47.342 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.365+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0881282c0 con 0x7fb0880ac5a0 2024-03-29T11:22:47.342 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.365+0000 7fb07dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.342 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.365+0000 7fb07dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.702+0000 7fb08f34b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.702+0000 7fb08f34b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.702+0000 7fb08f34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.702+0000 7fb08f34b640 1 -- 172.21.15.59:0/3156534660 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0881282c0 con 0x7fb0880ac5a0 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.703+0000 7fb07dffb640 1 -- 172.21.15.59:0/3156534660 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fb0881282c0 con 0x7fb0880ac5a0 2024-03-29T11:22:47.684 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.703+0000 7fb07dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:47.684 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.703+0000 7fb07dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:47.684 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:22:47.684 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:47.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 10 librados: watch_flush enter 2024-03-29T11:22:47.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 10 librados: watch_flush exit 2024-03-29T11:22:47.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T11:22:47.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 10 client.4330.objecter successfully canceled tick 2024-03-29T11:22:47.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 1 -- 172.21.15.59:0/3156534660 >> v1:172.21.15.172:6816/2420742773 conn(0x7fb06c06dc90 legacy=0x7fb06c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 10 monclient: shutdown 2024-03-29T11:22:47.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:47.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 1 -- 172.21.15.59:0/3156534660 >> v1:172.21.15.172:6789/0 conn(0x7fb0880ac5a0 legacy=0x7fb088146ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 1 -- 172.21.15.59:0/3156534660 shutdown_connections 2024-03-29T11:22:47.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 1 -- 172.21.15.59:0/3156534660 >> 172.21.15.59:0/3156534660 conn(0x7fb08806b660 msgr2=0x7fb08806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:47.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 1 -- 172.21.15.59:0/3156534660 shutdown_connections 2024-03-29T11:22:47.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.709+0000 7fb08f34b640 1 -- 172.21.15.59:0/3156534660 wait complete. 2024-03-29T11:22:47.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:47.710+0000 7fb08f34b640 1 librados: shutdown 2024-03-29T11:22:47.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:54.511 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:54.658 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.680+0000 7fd35ca95640 10 monclient: get_monmap_and_config 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.680+0000 7fd35ca95640 10 monclient: build_initial_monmap 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.680+0000 7fd35ca95640 10 monclient: monmap: 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:22:54.681323+0000 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:22:54.681323+0000 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:22:54.659 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:54.660 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.683+0000 7fd35ca95640 1 Processor -- start 2024-03-29T11:22:54.660 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.683+0000 7fd35ca95640 1 -- start start 2024-03-29T11:22:54.661 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.683+0000 7fd35ca95640 10 monclient: init 2024-03-29T11:22:54.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.685+0000 7fd35ca95640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:54.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.685+0000 7fd35ca95640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:54.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.685+0000 7fd35ca95640 10 monclient(hunting): picked mon.noname-b con 0x7fd3580acf60 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:54.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.685+0000 7fd35ca95640 10 monclient(hunting): picked mon.noname-c con 0x7fd3580ad5c0 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:54.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.685+0000 7fd35ca95640 10 monclient(hunting): picked mon.noname-a con 0x7fd35813a000 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:54.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.685+0000 7fd35ca95640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3580aec00 con 0x7fd35813a000 2024-03-29T11:22:54.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.685+0000 7fd35ca95640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35813daa0 con 0x7fd3580ad5c0 2024-03-29T11:22:54.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.685+0000 7fd35ca95640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3580ad9d0 con 0x7fd3580acf60 2024-03-29T11:22:54.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.685+0000 7fd35ca95640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:54.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.685+0000 7fd35ca95640 10 monclient(hunting): authenticate will time out at 1024.798886s 2024-03-29T11:22:54.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.688+0000 7fd35659c640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fd3580acf60 0x7fd35813ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:33217/0 (socket says 172.21.15.59:33217) 2024-03-29T11:22:54.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.688+0000 7fd35659c640 1 -- 172.21.15.59:0/3867172225 learned_addr learned my addr 172.21.15.59:0/3867172225 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:54.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.689+0000 7fd35559a640 1 -- 172.21.15.59:0/3867172225 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391196357 0 0) 0x7fd3580ad9d0 con 0x7fd3580acf60 2024-03-29T11:22:54.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.689+0000 7fd35559a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:54.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.689+0000 7fd35559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:54.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.689+0000 7fd35559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:54.667 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.689+0000 7fd35559a640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:22:54.667 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.689+0000 7fd35559a640 1 -- 172.21.15.59:0/3867172225 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd340003520 con 0x7fd3580acf60 2024-03-29T11:22:54.667 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.690+0000 7fd35559a640 1 -- 172.21.15.59:0/3867172225 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3198927744 0 0) 0x7fd340003520 con 0x7fd3580acf60 2024-03-29T11:22:54.667 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.690+0000 7fd35559a640 1 -- 172.21.15.59:0/3867172225 >> v1:172.21.15.172:6789/0 conn(0x7fd3580ad5c0 legacy=0x7fd3581398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:54.667 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.690+0000 7fd35559a640 1 -- 172.21.15.59:0/3867172225 >> v1:172.21.15.59:6789/0 conn(0x7fd35813a000 legacy=0x7fd35814b880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:54.668 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:54.668 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 1 monclient: found mon.noname-b 2024-03-29T11:22:54.668 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:54.668 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:22:54.668 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 1 -- 172.21.15.59:0/3867172225 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3580b4310 con 0x7fd3580acf60 2024-03-29T11:22:54.668 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 10 monclient: _finish_auth 0 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 1 -- 172.21.15.59:0/3867172225 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd348001690 con 0x7fd3580acf60 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 10 monclient: dump: 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:54.669 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35ca95640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35ca95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 1 -- 172.21.15.59:0/3867172225 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd348001860 con 0x7fd3580acf60 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 1 -- 172.21.15.59:0/3867172225 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd348001cb0 con 0x7fd3580acf60 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35559a640 10 monclient: dump: 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:54.670 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:54.671 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:54.671 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:54.671 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:54.671 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:54.671 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:54.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35ca95640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:54.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.691+0000 7fd35ca95640 4 set_mon_vals no callback set 2024-03-29T11:22:54.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.693+0000 7fd35ca95640 10 monclient: shutdown 2024-03-29T11:22:54.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.693+0000 7fd35ca95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:54.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.693+0000 7fd35ca95640 1 -- 172.21.15.59:0/3867172225 >> v1:172.21.15.59:6790/0 conn(0x7fd3580acf60 legacy=0x7fd35813ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:54.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.693+0000 7fd35ca95640 1 -- 172.21.15.59:0/3867172225 shutdown_connections 2024-03-29T11:22:54.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.693+0000 7fd35ca95640 1 -- 172.21.15.59:0/3867172225 >> 172.21.15.59:0/3867172225 conn(0x7fd3580b35e0 msgr2=0x7fd3580b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:54.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.695+0000 7fd35ca95640 1 -- 172.21.15.59:0/3867172225 shutdown_connections 2024-03-29T11:22:54.677 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.696+0000 7fd35ca95640 1 -- 172.21.15.59:0/3867172225 wait complete. 2024-03-29T11:22:54.677 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.697+0000 7fd35ca95640 10 monclient: build_initial_monmap 2024-03-29T11:22:54.677 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.697+0000 7fd35ca95640 10 monclient: monmap: 2024-03-29T11:22:54.677 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:22:54.677 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:54.677 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:22:54.677 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.698+0000 7fd35ca95640 1 librados: starting msgr at 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.698+0000 7fd35ca95640 1 librados: starting objecter 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.698+0000 7fd35ca95640 1 Processor -- start 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.698+0000 7fd35ca95640 1 -- start start 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.698+0000 7fd35ca95640 1 librados: setting wanted keys 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.698+0000 7fd35ca95640 1 librados: calling monclient init 2024-03-29T11:22:54.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.698+0000 7fd35ca95640 10 monclient: init 2024-03-29T11:22:54.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.698+0000 7fd35ca95640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:54.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.698+0000 7fd35ca95640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:54.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.699+0000 7fd35ca95640 10 monclient(hunting): picked mon.noname-c con 0x7fd3580ad5c0 addr v1:172.21.15.59:6790/0 2024-03-29T11:22:54.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.699+0000 7fd35ca95640 10 monclient(hunting): picked mon.noname-b con 0x7fd35813a000 addr v1:172.21.15.172:6789/0 2024-03-29T11:22:54.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.699+0000 7fd35ca95640 10 monclient(hunting): picked mon.noname-a con 0x7fd358142f70 addr v1:172.21.15.59:6789/0 2024-03-29T11:22:54.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.699+0000 7fd35ca95640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd358143ed0 con 0x7fd358142f70 2024-03-29T11:22:54.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.699+0000 7fd35ca95640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35813d330 con 0x7fd35813a000 2024-03-29T11:22:54.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.699+0000 7fd35ca95640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35813afb0 con 0x7fd3580ad5c0 2024-03-29T11:22:54.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.699+0000 7fd35ca95640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:54.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.699+0000 7fd35ca95640 10 monclient(hunting): authenticate will time out at 1024.812765s 2024-03-29T11:22:54.682 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.704+0000 7fd355d9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd35813a000 0x7fd358142860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:46273/0 (socket says 172.21.15.59:46273) 2024-03-29T11:22:54.682 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.704+0000 7fd355d9b640 1 -- 172.21.15.59:0/1595902450 learned_addr learned my addr 172.21.15.59:0/1595902450 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:22:54.682 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.704+0000 7fd346ffd640 10 client.?.objecter ms_handle_connect 0x7fd35813a000 2024-03-29T11:22:54.682 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.705+0000 7fd346ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:54.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.705+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010519306 0 0) 0x7fd35813d330 con 0x7fd35813a000 2024-03-29T11:22:54.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.705+0000 7fd346ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:54.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.705+0000 7fd346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:54.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.705+0000 7fd346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:54.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.705+0000 7fd346ffd640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:22:54.683 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.705+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd338004eb0 con 0x7fd35813a000 2024-03-29T11:22:54.685 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2716567814 0 0) 0x7fd338004eb0 con 0x7fd35813a000 2024-03-29T11:22:54.685 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd338005300 con 0x7fd35813a000 2024-03-29T11:22:54.685 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd34c001920 con 0x7fd35813a000 2024-03-29T11:22:54.685 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:54.685 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:54.685 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:54.685 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:54.686 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:54.686 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:54.686 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:54.686 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:54.686 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:54.686 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:54.686 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:54.686 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:54.686 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:54.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:54.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 375717853 0 0) 0x7fd338005300 con 0x7fd35813a000 2024-03-29T11:22:54.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 >> v1:172.21.15.59:6790/0 conn(0x7fd3580ad5c0 legacy=0x7fd3581437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:54.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 >> v1:172.21.15.59:6789/0 conn(0x7fd358142f70 legacy=0x7fd35813ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:54.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:54.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 1 monclient: found mon.b 2024-03-29T11:22:54.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:54.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:22:54.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.708+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35813d610 con 0x7fd35813a000 2024-03-29T11:22:54.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd346ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd346ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd346ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd346ffd640 10 client.?.objecter ms_handle_connect 0x7fd358142f70 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd346ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd35ca95640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd35ca95640 10 monclient: _renew_subs 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd35ca95640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd35ca95640 1 -- 172.21.15.59:0/1595902450 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35813b4d0 con 0x7fd35813a000 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd35ca95640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd35ca95640 10 monclient: _renew_subs 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd35ca95640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd35ca95640 1 -- 172.21.15.59:0/1595902450 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35813c2f0 con 0x7fd35813a000 2024-03-29T11:22:54.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.709+0000 7fd35ca95640 1 librados: init done 2024-03-29T11:22:54.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.713+0000 7fd344ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:54.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.713+0000 7fd344ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:54.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.713+0000 7fd344ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:22:54.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.713+0000 7fd344ff9640 1 -- 172.21.15.59:0/1595902450 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35813b140 con 0x7fd35813a000 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.713+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd34c001c40 con 0x7fd35813a000 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.713+0000 7fd346ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.713+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd34c001920 con 0x7fd35813a000 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.713+0000 7fd346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.713+0000 7fd346ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.713+0000 7fd346ffd640 10 monclient: dump: 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:22:54.691 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:22:54.692 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:22:54.692 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:22:54.692 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:22:54.692 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.713+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fd34c003920 con 0x7fd35813a000 2024-03-29T11:22:54.692 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.714+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fd34c084930 con 0x7fd35813a000 2024-03-29T11:22:54.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.714+0000 7fd346ffd640 10 client.4333.objecter ms_dispatch 0x7fd358000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:54.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.714+0000 7fd346ffd640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:54.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.714+0000 7fd346ffd640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:54.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.714+0000 7fd346ffd640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-29T11:22:54.695 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.715+0000 7fd347fff640 4 set_mon_vals no callback set 2024-03-29T11:22:54.695 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.717+0000 7fd35659c640 10 monclient: get_auth_request con 0x7fd33806e120 auth_method 0 2024-03-29T11:22:54.695 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.718+0000 7fd346ffd640 10 client.4333.objecter ms_handle_connect 0x7fd33806e120 2024-03-29T11:22:54.699 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.722+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd34c0500c0 con 0x7fd35813a000 2024-03-29T11:22:54.699 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.722+0000 7fd346ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:54.699 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:54.722+0000 7fd346ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:55.026 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.049+0000 7fd344ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.026 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.049+0000 7fd344ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.049+0000 7fd344ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:22:55.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.049+0000 7fd344ff9640 1 -- 172.21.15.59:0/1595902450 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd358075f10 con 0x7fd35813a000 2024-03-29T11:22:55.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.050+0000 7fd346ffd640 1 -- 172.21.15.59:0/1595902450 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fd34c0500c0 con 0x7fd35813a000 2024-03-29T11:22:55.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.050+0000 7fd346ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.050+0000 7fd346ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:55.029 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:22:55.030 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:55.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.054+0000 7fd344ff9640 10 librados: watch_flush enter 2024-03-29T11:22:55.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.054+0000 7fd344ff9640 10 librados: watch_flush exit 2024-03-29T11:22:55.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.054+0000 7fd344ff9640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-29T11:22:55.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.054+0000 7fd344ff9640 10 client.4333.objecter successfully canceled tick 2024-03-29T11:22:55.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.055+0000 7fd344ff9640 1 -- 172.21.15.59:0/1595902450 >> v1:172.21.15.172:6816/2420742773 conn(0x7fd33806e120 legacy=0x7fd3380705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.055+0000 7fd344ff9640 10 monclient: shutdown 2024-03-29T11:22:55.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.055+0000 7fd344ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:55.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.055+0000 7fd344ff9640 1 -- 172.21.15.59:0/1595902450 >> v1:172.21.15.172:6789/0 conn(0x7fd35813a000 legacy=0x7fd358142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.055+0000 7fd344ff9640 1 -- 172.21.15.59:0/1595902450 shutdown_connections 2024-03-29T11:22:55.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.055+0000 7fd344ff9640 1 -- 172.21.15.59:0/1595902450 >> 172.21.15.59:0/1595902450 conn(0x7fd3580b35e0 msgr2=0x7fd3580b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:55.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.055+0000 7fd344ff9640 1 -- 172.21.15.59:0/1595902450 shutdown_connections 2024-03-29T11:22:55.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.055+0000 7fd344ff9640 1 -- 172.21.15.59:0/1595902450 wait complete. 2024-03-29T11:22:55.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:22:55.055+0000 7fd344ff9640 1 librados: shutdown 2024-03-29T11:22:55.054 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:01.860 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:02.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.038+0000 7f515bef6640 10 monclient: get_monmap_and_config 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.038+0000 7f515bef6640 10 monclient: build_initial_monmap 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.038+0000 7f515bef6640 10 monclient: monmap: 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:23:02.040662+0000 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:23:02.040662+0000 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:23:02.018 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:02.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.039+0000 7f515bef6640 1 Processor -- start 2024-03-29T11:23:02.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 1 -- start start 2024-03-29T11:23:02.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 10 monclient: init 2024-03-29T11:23:02.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:02.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:02.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 10 monclient(hunting): picked mon.noname-a con 0x7f51541457f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:02.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 10 monclient(hunting): picked mon.noname-b con 0x7f515413f650 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:02.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 10 monclient(hunting): picked mon.noname-c con 0x7f51540a9d70 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:02.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51540aaef0 con 0x7f51541457f0 2024-03-29T11:23:02.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5154145bd0 con 0x7f51540a9d70 2024-03-29T11:23:02.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5154145eb0 con 0x7f515413f650 2024-03-29T11:23:02.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:02.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.040+0000 7f515bef6640 10 monclient(hunting): authenticate will time out at 1032.155670s 2024-03-29T11:23:02.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.041+0000 7f515a493640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f51540a9d70 0x7f515414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:43874/0 (socket says 172.21.15.59:43874) 2024-03-29T11:23:02.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.041+0000 7f515a493640 1 -- 172.21.15.59:0/2779993915 learned_addr learned my addr 172.21.15.59:0/2779993915 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:02.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 1 -- 172.21.15.59:0/2779993915 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768428789 0 0) 0x7f5154145bd0 con 0x7f51540a9d70 2024-03-29T11:23:02.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:23:02.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 1 -- 172.21.15.59:0/2779993915 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5140003520 con 0x7f51540a9d70 2024-03-29T11:23:02.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 1 -- 172.21.15.59:0/2779993915 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2079050353 0 0) 0x7f5140003520 con 0x7f51540a9d70 2024-03-29T11:23:02.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 1 -- 172.21.15.59:0/2779993915 >> v1:172.21.15.59:6790/0 conn(0x7f515413f650 legacy=0x7f51540b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 1 -- 172.21.15.59:0/2779993915 >> v1:172.21.15.59:6789/0 conn(0x7f51541457f0 legacy=0x7f515413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 10 monclient: _finish_hunting 0 2024-03-29T11:23:02.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 1 monclient: found mon.noname-c 2024-03-29T11:23:02.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:02.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:23:02.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 1 -- 172.21.15.59:0/2779993915 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5154146340 con 0x7f51540a9d70 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 10 monclient: _finish_auth 0 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 10 monclient: _check_auth_tickets 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 1 -- 172.21.15.59:0/2779993915 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5150001690 con 0x7f51540a9d70 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f5158c90640 10 monclient: dump: 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:02.023 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f515bef6640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.042+0000 7f515bef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f5158c90640 1 -- 172.21.15.59:0/2779993915 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51500017f0 con 0x7f51540a9d70 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f5158c90640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f5158c90640 1 -- 172.21.15.59:0/2779993915 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5150001c40 con 0x7f51540a9d70 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f515bef6640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f515bef6640 4 set_mon_vals no callback set 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f5158c90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f5158c90640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f5158c90640 10 monclient: dump: 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:02.024 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f515bef6640 10 monclient: shutdown 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f515bef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f515bef6640 1 -- 172.21.15.59:0/2779993915 >> v1:172.21.15.172:6789/0 conn(0x7f51540a9d70 legacy=0x7f515414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f515bef6640 1 -- 172.21.15.59:0/2779993915 shutdown_connections 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.043+0000 7f515bef6640 1 -- 172.21.15.59:0/2779993915 >> 172.21.15.59:0/2779993915 conn(0x7f5154066a00 msgr2=0x7f5154066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:02.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.044+0000 7f515bef6640 1 -- 172.21.15.59:0/2779993915 shutdown_connections 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.044+0000 7f515bef6640 1 -- 172.21.15.59:0/2779993915 wait complete. 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.044+0000 7f515bef6640 10 monclient: build_initial_monmap 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.044+0000 7f515bef6640 10 monclient: monmap: 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:23:02.028 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.045+0000 7f515bef6640 1 librados: starting msgr at 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.045+0000 7f515bef6640 1 librados: starting objecter 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.045+0000 7f515bef6640 1 Processor -- start 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.045+0000 7f515bef6640 1 -- start start 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.045+0000 7f515bef6640 1 librados: setting wanted keys 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.045+0000 7f515bef6640 1 librados: calling monclient init 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.045+0000 7f515bef6640 10 monclient: init 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f515bef6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f515bef6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f515bef6640 10 monclient(hunting): picked mon.noname-a con 0x7f51540a9d70 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f515bef6640 10 monclient(hunting): picked mon.noname-c con 0x7f515413f650 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f515bef6640 10 monclient(hunting): picked mon.noname-b con 0x7f51541457f0 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:02.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f515bef6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51540ad170 con 0x7f51540a9d70 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f515bef6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51540ab600 con 0x7f51541457f0 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f515bef6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5154142820 con 0x7f515413f650 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f515bef6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f515bef6640 10 monclient(hunting): authenticate will time out at 1032.161144s 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f5159491640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f515413f650 0x7f51541d7890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:34940/0 (socket says 172.21.15.59:34940) 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f5159c92640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f51540a9d70 0x7f51540abc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50950/0 (socket says 172.21.15.59:50950) 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f5159c92640 1 -- 172.21.15.59:0/3911395648 learned_addr learned my addr 172.21.15.59:0/3911395648 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f514a7fc640 10 client.?.objecter ms_handle_connect 0x7f51541457f0 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.046+0000 7f514a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.047+0000 7f514a7fc640 10 client.?.objecter ms_handle_connect 0x7f51540a9d70 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.047+0000 7f514a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.047+0000 7f514a7fc640 10 client.?.objecter ms_handle_connect 0x7f515413f650 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.047+0000 7f514a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.047+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56029054 0 0) 0x7f51540ab600 con 0x7f51541457f0 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.047+0000 7f514a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.047+0000 7f514a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.047+0000 7f514a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.047+0000 7f514a7fc640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.047+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5138004eb0 con 0x7f51541457f0 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.047+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158677790 0 0) 0x7f5154142820 con 0x7f515413f650 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51540ab600 con 0x7f515413f650 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785937459 0 0) 0x7f51540ad170 con 0x7f51540a9d70 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5154142820 con 0x7f51540a9d70 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 238385536 0 0) 0x7f5138004eb0 con 0x7f51541457f0 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51540ad170 con 0x7f51541457f0 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5150001b00 con 0x7f51541457f0 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): dump: 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.032 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 858726612 0 0) 0x7f51540ad170 con 0x7f51541457f0 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 >> v1:172.21.15.59:6790/0 conn(0x7f515413f650 legacy=0x7f51541d7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 >> v1:172.21.15.59:6789/0 conn(0x7f51540a9d70 legacy=0x7f51540abc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.048+0000 7f514a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 1 monclient: found mon.b 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5154142d00 con 0x7f51541457f0 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:02.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f515bef6640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f515bef6640 10 monclient: _renew_subs 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f515bef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f515bef6640 1 -- 172.21.15.59:0/3911395648 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51541d7fa0 con 0x7f51541457f0 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f515bef6640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f515bef6640 10 monclient: _renew_subs 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f515bef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f515bef6640 1 -- 172.21.15.59:0/3911395648 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f515413ff80 con 0x7f51541457f0 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f515bef6640 1 librados: init done 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5150001ca0 con 0x7f51541457f0 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514bfff640 4 set_mon_vals no callback set 2024-03-29T11:23:02.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5150004d90 con 0x7f51541457f0 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.049+0000 7f514a7fc640 10 monclient: dump: 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:02.035 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:02.036 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.051+0000 7f515bef6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.036 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.051+0000 7f515bef6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.036 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.051+0000 7f515bef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:02.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.051+0000 7f515bef6640 1 -- 172.21.15.59:0/3911395648 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c005080 con 0x7f51541457f0 2024-03-29T11:23:02.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.053+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f515001ad60 con 0x7f51541457f0 2024-03-29T11:23:02.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.054+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f5150084b00 con 0x7f51541457f0 2024-03-29T11:23:02.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.054+0000 7f514a7fc640 10 client.4339.objecter ms_dispatch 0x7f5154000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:02.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.054+0000 7f514a7fc640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:02.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.054+0000 7f514a7fc640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:02.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.054+0000 7f5159c92640 10 monclient: get_auth_request con 0x7f513806dc30 auth_method 0 2024-03-29T11:23:02.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.054+0000 7f514a7fc640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T11:23:02.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.054+0000 7f514a7fc640 10 client.4339.objecter ms_handle_connect 0x7f513806dc30 2024-03-29T11:23:02.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.058+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51500503e0 con 0x7f51541457f0 2024-03-29T11:23:02.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.058+0000 7f514a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.058+0000 7f514a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:02.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.452+0000 7f515bef6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:02.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.452+0000 7f515bef6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:02.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.452+0000 7f515bef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:02.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.452+0000 7f515bef6640 1 -- 172.21.15.59:0/3911395648 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f513c005330 con 0x7f51541457f0 2024-03-29T11:23:02.433 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:23:02.434 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:02.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.453+0000 7f514a7fc640 1 -- 172.21.15.59:0/3911395648 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f5150052310 con 0x7f51541457f0 2024-03-29T11:23:02.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.454+0000 7f514a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:02.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.454+0000 7f514a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:02.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.457+0000 7f515bef6640 10 librados: watch_flush enter 2024-03-29T11:23:02.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.457+0000 7f515bef6640 10 librados: watch_flush exit 2024-03-29T11:23:02.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.457+0000 7f515bef6640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T11:23:02.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.457+0000 7f515bef6640 10 client.4339.objecter successfully canceled tick 2024-03-29T11:23:02.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.457+0000 7f515bef6640 1 -- 172.21.15.59:0/3911395648 >> v1:172.21.15.172:6816/2420742773 conn(0x7f513806dc30 legacy=0x7f51380700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.457+0000 7f515bef6640 10 monclient: shutdown 2024-03-29T11:23:02.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.457+0000 7f515bef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:02.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.457+0000 7f515bef6640 1 -- 172.21.15.59:0/3911395648 >> v1:172.21.15.172:6789/0 conn(0x7f51541457f0 legacy=0x7f51541e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.457+0000 7f515bef6640 1 -- 172.21.15.59:0/3911395648 shutdown_connections 2024-03-29T11:23:02.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.457+0000 7f515bef6640 1 -- 172.21.15.59:0/3911395648 >> 172.21.15.59:0/3911395648 conn(0x7f5154066a00 msgr2=0x7f515413acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:02.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.458+0000 7f515bef6640 1 -- 172.21.15.59:0/3911395648 shutdown_connections 2024-03-29T11:23:02.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.458+0000 7f515bef6640 1 -- 172.21.15.59:0/3911395648 wait complete. 2024-03-29T11:23:02.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:02.458+0000 7f515bef6640 1 librados: shutdown 2024-03-29T11:23:02.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:09.259 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:09.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.425+0000 7f8729e82640 10 monclient: get_monmap_and_config 2024-03-29T11:23:09.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.425+0000 7f8729e82640 10 monclient: build_initial_monmap 2024-03-29T11:23:09.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.426+0000 7f8729e82640 10 monclient: monmap: 2024-03-29T11:23:09.405 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:09.405 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:09.405 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:23:09.427844+0000 2024-03-29T11:23:09.405 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:23:09.427844+0000 2024-03-29T11:23:09.405 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:09.405 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:09.405 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:09.405 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:23:09.406 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:23:09.406 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:09.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.427+0000 7f8729e82640 1 Processor -- start 2024-03-29T11:23:09.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.427+0000 7f8729e82640 1 -- start start 2024-03-29T11:23:09.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.427+0000 7f8729e82640 10 monclient: init 2024-03-29T11:23:09.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f8729e82640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:09.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f8729e82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:09.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f8729e82640 10 monclient(hunting): picked mon.noname-a con 0x7f872406ade0 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:09.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f8729e82640 10 monclient(hunting): picked mon.noname-c con 0x7f872406ed60 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:09.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f8729e82640 10 monclient(hunting): picked mon.noname-b con 0x7f8724072340 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:09.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f8729e82640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872406ff50 con 0x7f872406ade0 2024-03-29T11:23:09.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f8729e82640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8724070180 con 0x7f872406ed60 2024-03-29T11:23:09.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f8729e82640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872406f310 con 0x7f8724072340 2024-03-29T11:23:09.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f8729e82640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:09.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f8729e82640 10 monclient(hunting): authenticate will time out at 1039.543088s 2024-03-29T11:23:09.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f87237fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f872406ade0 0x7f87241499e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:56046/0 (socket says 172.21.15.59:56046) 2024-03-29T11:23:09.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.428+0000 7f87237fe640 1 -- 172.21.15.59:0/1285913498 learned_addr learned my addr 172.21.15.59:0/1285913498 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:09.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 1 -- 172.21.15.59:0/1285913498 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554136345 0 0) 0x7f872406f310 con 0x7f8724072340 2024-03-29T11:23:09.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:23:09.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 1 -- 172.21.15.59:0/1285913498 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f870c003520 con 0x7f8724072340 2024-03-29T11:23:09.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 1 -- 172.21.15.59:0/1285913498 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102625089 0 0) 0x7f872406ff50 con 0x7f872406ade0 2024-03-29T11:23:09.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.429+0000 7f87227fc640 1 -- 172.21.15.59:0/1285913498 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f872406f310 con 0x7f872406ade0 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 1 -- 172.21.15.59:0/1285913498 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3170588668 0 0) 0x7f872406f310 con 0x7f872406ade0 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 1 -- 172.21.15.59:0/1285913498 >> v1:172.21.15.59:6790/0 conn(0x7f8724072340 legacy=0x7f8724151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 1 -- 172.21.15.59:0/1285913498 >> v1:172.21.15.172:6789/0 conn(0x7f872406ed60 legacy=0x7f8724071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 1 monclient: found mon.noname-a 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 1 -- 172.21.15.59:0/1285913498 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87240aeed0 con 0x7f872406ade0 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:09.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 1 -- 172.21.15.59:0/1285913498 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8714001670 con 0x7f872406ade0 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f8729e82640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f8729e82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 10 monclient: dump: 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:09.411 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:09.412 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:09.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f8729e82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:09.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 1 -- 172.21.15.59:0/1285913498 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87140017d0 con 0x7f872406ade0 2024-03-29T11:23:09.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:09.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 1 -- 172.21.15.59:0/1285913498 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8714001c20 con 0x7f872406ade0 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f87227fc640 10 monclient: dump: 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:09.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f8729e82640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.430+0000 7f8729e82640 4 set_mon_vals no callback set 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.431+0000 7f8729e82640 10 monclient: shutdown 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.431+0000 7f8729e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.431+0000 7f8729e82640 1 -- 172.21.15.59:0/1285913498 >> v1:172.21.15.59:6789/0 conn(0x7f872406ade0 legacy=0x7f87241499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.431+0000 7f8729e82640 1 -- 172.21.15.59:0/1285913498 shutdown_connections 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.431+0000 7f8729e82640 1 -- 172.21.15.59:0/1285913498 >> 172.21.15.59:0/1285913498 conn(0x7f87240b4930 msgr2=0x7f87240b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.431+0000 7f8729e82640 1 -- 172.21.15.59:0/1285913498 shutdown_connections 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.431+0000 7f8729e82640 1 -- 172.21.15.59:0/1285913498 wait complete. 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.432+0000 7f8729e82640 10 monclient: build_initial_monmap 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.432+0000 7f8729e82640 10 monclient: monmap: 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:23:09.422 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.433+0000 7f8729e82640 1 librados: starting msgr at 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.433+0000 7f8729e82640 1 librados: starting objecter 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.433+0000 7f8729e82640 1 Processor -- start 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.433+0000 7f8729e82640 1 -- start start 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.433+0000 7f8729e82640 1 librados: setting wanted keys 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.433+0000 7f8729e82640 1 librados: calling monclient init 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.433+0000 7f8729e82640 10 monclient: init 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f8729e82640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:09.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f8729e82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f8729e82640 10 monclient(hunting): picked mon.noname-a con 0x7f872406ade0 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f8729e82640 10 monclient(hunting): picked mon.noname-c con 0x7f872406ed60 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f8729e82640 10 monclient(hunting): picked mon.noname-b con 0x7f8724072340 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f8729e82640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87240ad130 con 0x7f872406ade0 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f8729e82640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87240abb00 con 0x7f8724072340 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f8729e82640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87240ac370 con 0x7f872406ed60 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f8729e82640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f8729e82640 10 monclient(hunting): authenticate will time out at 1039.549328s 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f87237fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f872406ade0 0x7f87240aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:56286/0 (socket says 172.21.15.59:56286) 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.434+0000 7f87237fe640 1 -- 172.21.15.59:0/3087223112 learned_addr learned my addr 172.21.15.59:0/3087223112 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 client.?.objecter ms_handle_connect 0x7f8724072340 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 client.?.objecter ms_handle_connect 0x7f872406ade0 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 client.?.objecter ms_handle_connect 0x7f872406ed60 2024-03-29T11:23:09.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239943635 0 0) 0x7f87240abb00 con 0x7f8724072340 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8710004eb0 con 0x7f8724072340 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114632032 0 0) 0x7f87240ac370 con 0x7f872406ed60 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87240abb00 con 0x7f872406ed60 2024-03-29T11:23:09.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207738136 0 0) 0x7f87240ad130 con 0x7f872406ade0 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.435+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87240ac370 con 0x7f872406ade0 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3502796128 0 0) 0x7f8710004eb0 con 0x7f8724072340 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87240ad130 con 0x7f8724072340 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f871c001770 con 0x7f8724072340 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 10 monclient(hunting): dump: 2024-03-29T11:23:09.426 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 370593703 0 0) 0x7f87240ac370 con 0x7f872406ade0 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8710004eb0 con 0x7f872406ade0 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8714001b60 con 0x7f872406ade0 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:09.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 10 monclient(hunting): dump: 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2583238546 0 0) 0x7f87240abb00 con 0x7f872406ed60 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87240ac370 con 0x7f872406ed60 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8718001c30 con 0x7f872406ed60 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:09.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.436+0000 7f870bfff640 10 monclient(hunting): dump: 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3082064054 0 0) 0x7f8710004eb0 con 0x7f872406ade0 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 >> v1:172.21.15.59:6790/0 conn(0x7f872406ed60 legacy=0x7f87241db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 >> v1:172.21.15.172:6789/0 conn(0x7f8724072340 legacy=0x7f8724140ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f870bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f870bfff640 1 monclient: found mon.a 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f870bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f870bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87240ac500 con 0x7f872406ade0 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f870bfff640 10 monclient: _finish_auth 0 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f870bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f870bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f8729e82640 5 monclient: authenticate success, global_id 4365 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f8729e82640 10 monclient: _renew_subs 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f8729e82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f8729e82640 1 -- 172.21.15.59:0/3087223112 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87240ae5f0 con 0x7f872406ade0 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f8729e82640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f8729e82640 10 monclient: _renew_subs 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f8729e82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:09.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f8729e82640 1 -- 172.21.15.59:0/3087223112 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8724141bb0 con 0x7f872406ade0 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.437+0000 7f8729e82640 1 librados: init done 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.438+0000 7f8729e82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.438+0000 7f8729e82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.438+0000 7f8729e82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.438+0000 7f8729e82640 1 -- 172.21.15.59:0/3087223112 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8724075f10 con 0x7f872406ade0 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.439+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8714004b00 con 0x7f872406ade0 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.439+0000 7f870bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.439+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8714004ed0 con 0x7f872406ade0 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.439+0000 7f8721ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.440+0000 7f870bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.440+0000 7f870bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.440+0000 7f870bfff640 10 monclient: dump: 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.431 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:09.432 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.432 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:09.432 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:09.432 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:09.432 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:09.432 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:09.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.440+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f87140050f0 con 0x7f872406ade0 2024-03-29T11:23:09.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.442+0000 7f8722ffd640 10 monclient: get_auth_request con 0x7f871006e0e0 auth_method 0 2024-03-29T11:23:09.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.443+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f8714084b30 con 0x7f872406ade0 2024-03-29T11:23:09.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.443+0000 7f870bfff640 10 client.4365.objecter ms_dispatch 0x7f8724000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:09.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.443+0000 7f870bfff640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:09.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.443+0000 7f870bfff640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:09.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.443+0000 7f870bfff640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-29T11:23:09.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.444+0000 7f870bfff640 10 client.4365.objecter ms_handle_connect 0x7f871006e0e0 2024-03-29T11:23:09.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.450+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8714050410 con 0x7f872406ade0 2024-03-29T11:23:09.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.450+0000 7f870bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:09.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.450+0000 7f870bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:09.846 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.867+0000 7f8729e82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:09.846 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.867+0000 7f8729e82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:09.846 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.867+0000 7f8729e82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:09.846 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.867+0000 7f8729e82640 1 -- 172.21.15.59:0/3087223112 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f872406b1c0 con 0x7f872406ade0 2024-03-29T11:23:09.847 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:23:09.847 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:09.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.868+0000 7f870bfff640 1 -- 172.21.15.59:0/3087223112 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f8714052340 con 0x7f872406ade0 2024-03-29T11:23:09.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.868+0000 7f870bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:09.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.868+0000 7f870bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:09.850 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.871+0000 7f8729e82640 10 librados: watch_flush enter 2024-03-29T11:23:09.850 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.871+0000 7f8729e82640 10 librados: watch_flush exit 2024-03-29T11:23:09.850 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.871+0000 7f8729e82640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-29T11:23:09.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.871+0000 7f8729e82640 10 client.4365.objecter successfully canceled tick 2024-03-29T11:23:09.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.871+0000 7f8729e82640 1 -- 172.21.15.59:0/3087223112 >> v1:172.21.15.172:6816/2420742773 conn(0x7f871006e0e0 legacy=0x7f87100705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.871+0000 7f8729e82640 10 monclient: shutdown 2024-03-29T11:23:09.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.871+0000 7f8729e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:09.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.871+0000 7f8729e82640 1 -- 172.21.15.59:0/3087223112 >> v1:172.21.15.59:6789/0 conn(0x7f872406ade0 legacy=0x7f87240aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.872+0000 7f8729e82640 1 -- 172.21.15.59:0/3087223112 shutdown_connections 2024-03-29T11:23:09.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.872+0000 7f8729e82640 1 -- 172.21.15.59:0/3087223112 >> 172.21.15.59:0/3087223112 conn(0x7f87240b4930 msgr2=0x7f87240b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:09.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.872+0000 7f8729e82640 1 -- 172.21.15.59:0/3087223112 shutdown_connections 2024-03-29T11:23:09.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.872+0000 7f8729e82640 1 -- 172.21.15.59:0/3087223112 wait complete. 2024-03-29T11:23:09.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:09.872+0000 7f8729e82640 1 librados: shutdown 2024-03-29T11:23:09.866 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:16.672 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:16.805 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.825+0000 7f2cbb315640 10 monclient: get_monmap_and_config 2024-03-29T11:23:16.805 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.825+0000 7f2cbb315640 10 monclient: build_initial_monmap 2024-03-29T11:23:16.805 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.825+0000 7f2cbb315640 10 monclient: monmap: 2024-03-29T11:23:16.805 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:16.806 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:16.806 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:23:16.827645+0000 2024-03-29T11:23:16.806 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:23:16.827645+0000 2024-03-29T11:23:16.806 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:16.806 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:16.806 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:16.806 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:23:16.806 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:23:16.806 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:16.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.826+0000 7f2cbb315640 1 Processor -- start 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.826+0000 7f2cbb315640 1 -- start start 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.826+0000 7f2cbb315640 10 monclient: init 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cbb315640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cbb315640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cbb315640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb41422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cbb315640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb4142690 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cbb315640 10 monclient(hunting): picked mon.noname-b con 0x7f2cb40b17d0 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cbb315640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb4142c80 con 0x7f2cb4142690 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cbb315640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40aae40 con 0x7f2cb41422b0 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cbb315640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb413fbd0 con 0x7f2cb40b17d0 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cbb315640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cbb315640 10 monclient(hunting): authenticate will time out at 1046.942193s 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cb90b1640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2cb41422b0 0x7f2cb413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:46356/0 (socket says 172.21.15.59:46356) 2024-03-29T11:23:16.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.827+0000 7f2cb90b1640 1 -- 172.21.15.59:0/3505700764 learned_addr learned my addr 172.21.15.59:0/3505700764 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 1 -- 172.21.15.59:0/3505700764 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514342317 0 0) 0x7f2cb40aae40 con 0x7f2cb41422b0 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 1 -- 172.21.15.59:0/3505700764 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca0003520 con 0x7f2cb41422b0 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 1 -- 172.21.15.59:0/3505700764 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098656033 0 0) 0x7f2cb413fbd0 con 0x7f2cb40b17d0 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 1 -- 172.21.15.59:0/3505700764 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb40aae40 con 0x7f2cb40b17d0 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 1 -- 172.21.15.59:0/3505700764 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3253620401 0 0) 0x7f2cb40aae40 con 0x7f2cb40b17d0 2024-03-29T11:23:16.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 1 -- 172.21.15.59:0/3505700764 >> v1:172.21.15.172:6789/0 conn(0x7f2cb41422b0 legacy=0x7f2cb413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.828+0000 7f2cabfff640 1 -- 172.21.15.59:0/3505700764 >> v1:172.21.15.59:6789/0 conn(0x7f2cb4142690 legacy=0x7f2cb40b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:16.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:16.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 1 monclient: found mon.noname-b 2024-03-29T11:23:16.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 1 -- 172.21.15.59:0/3505700764 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb4151940 con 0x7f2cb40b17d0 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 10 monclient: _finish_auth 0 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 1 -- 172.21.15.59:0/3505700764 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2cac0019f0 con 0x7f2cb40b17d0 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 10 monclient: dump: 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:16.812 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cbb315640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cbb315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 1 -- 172.21.15.59:0/3505700764 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cac001b50 con 0x7f2cb40b17d0 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 1 -- 172.21.15.59:0/3505700764 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2cac001f20 con 0x7f2cb40b17d0 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:16.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cabfff640 10 monclient: dump: 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cbb315640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:16.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.831+0000 7f2cbb315640 4 set_mon_vals no callback set 2024-03-29T11:23:16.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.832+0000 7f2cbb315640 10 monclient: shutdown 2024-03-29T11:23:16.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.832+0000 7f2cbb315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:16.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.832+0000 7f2cbb315640 1 -- 172.21.15.59:0/3505700764 >> v1:172.21.15.59:6790/0 conn(0x7f2cb40b17d0 legacy=0x7f2cb40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.833+0000 7f2cbb315640 1 -- 172.21.15.59:0/3505700764 shutdown_connections 2024-03-29T11:23:16.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.833+0000 7f2cbb315640 1 -- 172.21.15.59:0/3505700764 >> 172.21.15.59:0/3505700764 conn(0x7f2cb4066a00 msgr2=0x7f2cb4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:16.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.833+0000 7f2cbb315640 1 -- 172.21.15.59:0/3505700764 shutdown_connections 2024-03-29T11:23:16.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.833+0000 7f2cbb315640 1 -- 172.21.15.59:0/3505700764 wait complete. 2024-03-29T11:23:16.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.834+0000 7f2cbb315640 10 monclient: build_initial_monmap 2024-03-29T11:23:16.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.834+0000 7f2cbb315640 10 monclient: monmap: 2024-03-29T11:23:16.821 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:16.821 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:16.821 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.834+0000 7f2cbb315640 1 librados: starting msgr at 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.834+0000 7f2cbb315640 1 librados: starting objecter 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.834+0000 7f2cbb315640 1 Processor -- start 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.834+0000 7f2cbb315640 1 -- start start 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.834+0000 7f2cbb315640 1 librados: setting wanted keys 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.834+0000 7f2cbb315640 1 librados: calling monclient init 2024-03-29T11:23:16.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.834+0000 7f2cbb315640 10 monclient: init 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.835+0000 7f2cbb315640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.835+0000 7f2cbb315640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.835+0000 7f2cbb315640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb40b17d0 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.835+0000 7f2cbb315640 10 monclient(hunting): picked mon.noname-b con 0x7f2cb41422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.835+0000 7f2cbb315640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb4142690 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.835+0000 7f2cbb315640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40ac8c0 con 0x7f2cb4142690 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.835+0000 7f2cbb315640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40acb30 con 0x7f2cb41422b0 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2cbb315640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40ace10 con 0x7f2cb40b17d0 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2cbb315640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2cbb315640 10 monclient(hunting): authenticate will time out at 1046.950830s 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2cb88b0640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2cb41422b0 0x7f2cb412e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:47486/0 (socket says 172.21.15.59:47486) 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2cb88b0640 1 -- 172.21.15.59:0/599519139 learned_addr learned my addr 172.21.15.59:0/599519139 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2ca9ffb640 10 client.?.objecter ms_handle_connect 0x7f2cb40b17d0 2024-03-29T11:23:16.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2ca9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211806602 0 0) 0x7f2cb40ace10 con 0x7f2cb40b17d0 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2ca9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2ca9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2ca9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2ca9ffb640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.836+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c98004eb0 con 0x7f2cb40b17d0 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3165840950 0 0) 0x7f2c98004eb0 con 0x7f2cb40b17d0 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cb40ace10 con 0x7f2cb40b17d0 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2cb0001950 con 0x7f2cb40b17d0 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:16.824 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 10 client.?.objecter ms_handle_connect 0x7f2cb41422b0 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 113013116 0 0) 0x7f2cb40ace10 con 0x7f2cb40b17d0 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 >> v1:172.21.15.172:6789/0 conn(0x7f2cb41422b0 legacy=0x7f2cb412e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:16.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.837+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 >> v1:172.21.15.59:6789/0 conn(0x7f2cb4142690 legacy=0x7f2cb41e1c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 1 monclient: found mon.c 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb4131d70 con 0x7f2cb40b17d0 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2cbb315640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2cbb315640 10 monclient: _renew_subs 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2cbb315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2cbb315640 1 -- 172.21.15.59:0/599519139 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb41327a0 con 0x7f2cb40b17d0 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cb00047e0 con 0x7f2cb40b17d0 2024-03-29T11:23:16.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2cb0004bf0 con 0x7f2cb40b17d0 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2ca9ffb640 10 monclient: dump: 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:16.827 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:16.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.838+0000 7f2cab7fe640 4 set_mon_vals no callback set 2024-03-29T11:23:16.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.839+0000 7f2cbb315640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:16.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.839+0000 7f2cbb315640 10 monclient: _renew_subs 2024-03-29T11:23:16.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.839+0000 7f2cbb315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:23:16.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.839+0000 7f2cbb315640 1 -- 172.21.15.59:0/599519139 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb40aba90 con 0x7f2cb40b17d0 2024-03-29T11:23:16.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.839+0000 7f2cbb315640 1 librados: init done 2024-03-29T11:23:16.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.841+0000 7f2cbb315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:16.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.841+0000 7f2cbb315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:16.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.841+0000 7f2cbb315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:23:16.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.841+0000 7f2cbb315640 1 -- 172.21.15.59:0/599519139 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb40abde0 con 0x7f2cb40b17d0 2024-03-29T11:23:16.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.841+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f2cb0004d50 con 0x7f2cb40b17d0 2024-03-29T11:23:16.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.842+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f2cb0084b30 con 0x7f2cb40b17d0 2024-03-29T11:23:16.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.842+0000 7f2ca9ffb640 10 client.4352.objecter ms_dispatch 0x7f2cb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:16.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.842+0000 7f2ca9ffb640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:16.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.842+0000 7f2ca9ffb640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:16.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.842+0000 7f2ca9ffb640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T11:23:16.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.842+0000 7f2cb88b0640 10 monclient: get_auth_request con 0x7f2c9806dc90 auth_method 0 2024-03-29T11:23:16.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.843+0000 7f2ca9ffb640 10 client.4352.objecter ms_handle_connect 0x7f2c9806dc90 2024-03-29T11:23:16.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.848+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2cb0050410 con 0x7f2cb40b17d0 2024-03-29T11:23:16.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.849+0000 7f2ca9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:16.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:16.849+0000 7f2ca9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:17.142 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.163+0000 7f2cbb315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.143 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.163+0000 7f2cbb315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.143 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.163+0000 7f2cbb315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:23:17.143 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.163+0000 7f2cbb315640 1 -- 172.21.15.59:0/599519139 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cb4131f00 con 0x7f2cb40b17d0 2024-03-29T11:23:17.144 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.164+0000 7f2ca9ffb640 1 -- 172.21.15.59:0/599519139 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f2cb0052340 con 0x7f2cb40b17d0 2024-03-29T11:23:17.144 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.164+0000 7f2ca9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.144 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.164+0000 7f2ca9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:17.144 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:23:17.144 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.167+0000 7f2cbb315640 10 librados: watch_flush enter 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.167+0000 7f2cbb315640 10 librados: watch_flush exit 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.167+0000 7f2cbb315640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.167+0000 7f2cbb315640 10 client.4352.objecter successfully canceled tick 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.167+0000 7f2cbb315640 1 -- 172.21.15.59:0/599519139 >> v1:172.21.15.172:6816/2420742773 conn(0x7f2c9806dc90 legacy=0x7f2c98070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.167+0000 7f2cbb315640 10 monclient: shutdown 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.167+0000 7f2cbb315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.167+0000 7f2cbb315640 1 -- 172.21.15.59:0/599519139 >> v1:172.21.15.59:6790/0 conn(0x7f2cb40b17d0 legacy=0x7f2cb4150d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.167+0000 7f2cbb315640 1 -- 172.21.15.59:0/599519139 shutdown_connections 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.167+0000 7f2cbb315640 1 -- 172.21.15.59:0/599519139 >> 172.21.15.59:0/599519139 conn(0x7f2cb4066a00 msgr2=0x7f2cb40b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.168+0000 7f2cbb315640 1 -- 172.21.15.59:0/599519139 shutdown_connections 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.168+0000 7f2cbb315640 1 -- 172.21.15.59:0/599519139 wait complete. 2024-03-29T11:23:17.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:17.168+0000 7f2cbb315640 1 librados: shutdown 2024-03-29T11:23:17.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:23.969 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:24.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.143+0000 7f088ef92640 10 monclient: get_monmap_and_config 2024-03-29T11:23:24.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.143+0000 7f088ef92640 10 monclient: build_initial_monmap 2024-03-29T11:23:24.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.143+0000 7f088ef92640 10 monclient: monmap: 2024-03-29T11:23:24.122 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:24.122 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:24.122 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:23:24.143931+0000 2024-03-29T11:23:24.123 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:23:24.143931+0000 2024-03-29T11:23:24.123 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:24.123 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:24.123 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:24.123 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:23:24.123 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:23:24.123 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:24.123 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.144+0000 7f088ef92640 1 Processor -- start 2024-03-29T11:23:24.123 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.144+0000 7f088ef92640 1 -- start start 2024-03-29T11:23:24.123 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.144+0000 7f088ef92640 10 monclient: init 2024-03-29T11:23:24.124 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.145+0000 7f088ef92640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:24.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.145+0000 7f088ef92640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:24.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.145+0000 7f088ef92640 10 monclient(hunting): picked mon.noname-c con 0x7f08880b4820 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:24.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.145+0000 7f088ef92640 10 monclient(hunting): picked mon.noname-b con 0x7f08880a9ce0 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:24.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.145+0000 7f088ef92640 10 monclient(hunting): picked mon.noname-a con 0x7f08880b3cd0 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:24.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.145+0000 7f088ef92640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08880b4180 con 0x7f08880b3cd0 2024-03-29T11:23:24.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.145+0000 7f088ef92640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08880aaf60 con 0x7f08880b4820 2024-03-29T11:23:24.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.145+0000 7f088ef92640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088813c200 con 0x7f08880a9ce0 2024-03-29T11:23:24.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.145+0000 7f088ef92640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:24.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.145+0000 7f088ef92640 10 monclient(hunting): authenticate will time out at 1054.259482s 2024-03-29T11:23:24.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.146+0000 7f088d78f640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f08880a9ce0 0x7f0888145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:50640/0 (socket says 172.21.15.59:50640) 2024-03-29T11:23:24.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.146+0000 7f088d78f640 1 -- 172.21.15.59:0/1529942238 learned_addr learned my addr 172.21.15.59:0/1529942238 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.148+0000 7f088cf8e640 1 -- 172.21.15.59:0/1529942238 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174070975 0 0) 0x7f088813c200 con 0x7f08880a9ce0 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.148+0000 7f088cf8e640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.148+0000 7f088cf8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.148+0000 7f088cf8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.148+0000 7f088cf8e640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.148+0000 7f088cf8e640 1 -- 172.21.15.59:0/1529942238 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0874003520 con 0x7f08880a9ce0 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.148+0000 7f088cf8e640 1 -- 172.21.15.59:0/1529942238 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1756610974 0 0) 0x7f0874003520 con 0x7f08880a9ce0 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.148+0000 7f088cf8e640 1 -- 172.21.15.59:0/1529942238 >> v1:172.21.15.172:6789/0 conn(0x7f08880b4820 legacy=0x7f088813baf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 1 -- 172.21.15.59:0/1529942238 >> v1:172.21.15.59:6789/0 conn(0x7f08880b3cd0 legacy=0x7f088814d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 10 monclient: _finish_hunting 0 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 1 monclient: found mon.noname-b 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 1 -- 172.21.15.59:0/1529942238 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0888145e80 con 0x7f08880a9ce0 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 10 monclient: _finish_auth 0 2024-03-29T11:23:24.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 10 monclient: _check_auth_tickets 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 1 -- 172.21.15.59:0/1529942238 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0878001690 con 0x7f08880a9ce0 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088cf8e640 10 monclient: dump: 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:24.130 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:24.131 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:24.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088ef92640 5 monclient: authenticate success, global_id 4355 2024-03-29T11:23:24.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.149+0000 7f088ef92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:24.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.150+0000 7f088cf8e640 1 -- 172.21.15.59:0/1529942238 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0878001860 con 0x7f08880a9ce0 2024-03-29T11:23:24.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.150+0000 7f088cf8e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:24.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.150+0000 7f088ef92640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:24.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.150+0000 7f088ef92640 4 set_mon_vals no callback set 2024-03-29T11:23:24.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.151+0000 7f088ef92640 10 monclient: shutdown 2024-03-29T11:23:24.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.151+0000 7f088ef92640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:24.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.151+0000 7f088ef92640 1 -- 172.21.15.59:0/1529942238 >> v1:172.21.15.59:6790/0 conn(0x7f08880a9ce0 legacy=0x7f0888145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.152+0000 7f088ef92640 1 -- 172.21.15.59:0/1529942238 shutdown_connections 2024-03-29T11:23:24.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.152+0000 7f088ef92640 1 -- 172.21.15.59:0/1529942238 >> 172.21.15.59:0/1529942238 conn(0x7f088806d070 msgr2=0x7f0888066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:24.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.152+0000 7f088ef92640 1 -- 172.21.15.59:0/1529942238 shutdown_connections 2024-03-29T11:23:24.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.152+0000 7f088ef92640 1 -- 172.21.15.59:0/1529942238 wait complete. 2024-03-29T11:23:24.133 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.154+0000 7f088ef92640 10 monclient: build_initial_monmap 2024-03-29T11:23:24.133 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.154+0000 7f088ef92640 10 monclient: monmap: 2024-03-29T11:23:24.133 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:24.133 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:24.133 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:23:24.133 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:23:24.134 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:24.134 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:24.134 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:24.134 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:23:24.134 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:23:24.134 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:24.134 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.154+0000 7f088ef92640 1 librados: starting msgr at 2024-03-29T11:23:24.134 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.154+0000 7f088ef92640 1 librados: starting objecter 2024-03-29T11:23:24.135 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.155+0000 7f088ef92640 1 Processor -- start 2024-03-29T11:23:24.137 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.157+0000 7f088ef92640 1 -- start start 2024-03-29T11:23:24.137 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.157+0000 7f088ef92640 1 librados: setting wanted keys 2024-03-29T11:23:24.137 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.157+0000 7f088ef92640 1 librados: calling monclient init 2024-03-29T11:23:24.137 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.157+0000 7f088ef92640 10 monclient: init 2024-03-29T11:23:24.140 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.157+0000 7f088ef92640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:24.140 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.158+0000 7f088ef92640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:24.140 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.158+0000 7f088ef92640 10 monclient(hunting): picked mon.noname-a con 0x7f08880a9ce0 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:24.140 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.158+0000 7f088ef92640 10 monclient(hunting): picked mon.noname-b con 0x7f08880b3cd0 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:24.140 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.158+0000 7f088ef92640 10 monclient(hunting): picked mon.noname-c con 0x7f08880b4820 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:24.140 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.158+0000 7f088ef92640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08881db120 con 0x7f08880a9ce0 2024-03-29T11:23:24.141 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.158+0000 7f088ef92640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08881dbff0 con 0x7f08880b3cd0 2024-03-29T11:23:24.141 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.158+0000 7f088ef92640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0888071180 con 0x7f08880b4820 2024-03-29T11:23:24.141 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.158+0000 7f088ef92640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:24.141 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.158+0000 7f088ef92640 10 monclient(hunting): authenticate will time out at 1054.272245s 2024-03-29T11:23:24.141 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.159+0000 7f088e791640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f08880b4820 0x7f08881e5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:46914/0 (socket says 172.21.15.59:46914) 2024-03-29T11:23:24.141 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.159+0000 7f088e791640 1 -- 172.21.15.59:0/2994758201 learned_addr learned my addr 172.21.15.59:0/2994758201 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:24.141 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.159+0000 7f087e7fc640 10 client.?.objecter ms_handle_connect 0x7f08880b4820 2024-03-29T11:23:24.141 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.159+0000 7f087e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.162+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395292490 0 0) 0x7f0888071180 con 0x7f08880b4820 2024-03-29T11:23:24.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.162+0000 7f087e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.162+0000 7f087e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.162+0000 7f087e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.162+0000 7f087e7fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:23:24.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.162+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c004eb0 con 0x7f08880b4820 2024-03-29T11:23:24.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.162+0000 7f087e7fc640 10 client.?.objecter ms_handle_connect 0x7f08880b3cd0 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.162+0000 7f087e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.162+0000 7f087e7fc640 10 client.?.objecter ms_handle_connect 0x7f08880a9ce0 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 546059770 0 0) 0x7f086c004eb0 con 0x7f08880b4820 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0888071180 con 0x7f08880b4820 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134230085 0 0) 0x7f08881dbff0 con 0x7f08880b3cd0 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c004eb0 con 0x7f08880b3cd0 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0880001c60 con 0x7f08880b4820 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.163+0000 7f087e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 10 monclient(hunting): dump: 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2432686754 0 0) 0x7f0888071180 con 0x7f08880b4820 2024-03-29T11:23:24.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 >> v1:172.21.15.172:6789/0 conn(0x7f08880b3cd0 legacy=0x7f08881db8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 >> v1:172.21.15.59:6789/0 conn(0x7f08880a9ce0 legacy=0x7f0888071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 1 monclient: found mon.c 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08881db3c0 con 0x7f08880b4820 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f087e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f088ef92640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f088ef92640 10 monclient: _renew_subs 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f088ef92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f088ef92640 1 -- 172.21.15.59:0/2994758201 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08880b0900 con 0x7f08880b4820 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f088ef92640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f088ef92640 10 monclient: _renew_subs 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f088ef92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:23:24.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f088ef92640 1 -- 172.21.15.59:0/2994758201 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f088806ed00 con 0x7f08880b4820 2024-03-29T11:23:24.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.164+0000 7f088ef92640 1 librados: init done 2024-03-29T11:23:24.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.167+0000 7f088ef92640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.167+0000 7f088ef92640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.167+0000 7f088ef92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:23:24.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.167+0000 7f088ef92640 1 -- 172.21.15.59:0/2994758201 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088804f730 con 0x7f08880b4820 2024-03-29T11:23:24.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.170+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0880001dc0 con 0x7f08880b4820 2024-03-29T11:23:24.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.170+0000 7f087e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:24.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.170+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0880003c20 con 0x7f08880b4820 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.170+0000 7f087e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.170+0000 7f087e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.170+0000 7f087e7fc640 10 monclient: dump: 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:24.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.171+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f0880003e40 con 0x7f08880b4820 2024-03-29T11:23:24.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.172+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f0880084d00 con 0x7f08880b4820 2024-03-29T11:23:24.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.172+0000 7f088df90640 10 monclient: get_auth_request con 0x7f086c06de90 auth_method 0 2024-03-29T11:23:24.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.172+0000 7f087e7fc640 10 client.4358.objecter ms_dispatch 0x7f0888000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:24.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.172+0000 7f087e7fc640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:24.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.172+0000 7f087e7fc640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:24.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.172+0000 7f087e7fc640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-29T11:23:24.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.172+0000 7f087e7fc640 10 client.4358.objecter ms_handle_connect 0x7f086c06de90 2024-03-29T11:23:24.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.172+0000 7f087f7fe640 4 set_mon_vals no callback set 2024-03-29T11:23:24.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.179+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0880050620 con 0x7f08880b4820 2024-03-29T11:23:24.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.179+0000 7f087e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.179+0000 7f087e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:24.503 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.523+0000 7f088ef92640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:24.503 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.524+0000 7f088ef92640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:24.503 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.524+0000 7f088ef92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:23:24.503 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.524+0000 7f088ef92640 1 -- 172.21.15.59:0/2994758201 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08880b6b10 con 0x7f08880b4820 2024-03-29T11:23:24.504 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.524+0000 7f087e7fc640 1 -- 172.21.15.59:0/2994758201 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f0880052510 con 0x7f08880b4820 2024-03-29T11:23:24.504 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.525+0000 7f087e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:24.504 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.525+0000 7f087e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:24.504 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:23:24.504 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:24.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.528+0000 7f088ef92640 10 librados: watch_flush enter 2024-03-29T11:23:24.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.530+0000 7f088ef92640 10 librados: watch_flush exit 2024-03-29T11:23:24.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.530+0000 7f088ef92640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-29T11:23:24.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.530+0000 7f088ef92640 10 client.4358.objecter successfully canceled tick 2024-03-29T11:23:24.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.530+0000 7f088ef92640 1 -- 172.21.15.59:0/2994758201 >> v1:172.21.15.172:6816/2420742773 conn(0x7f086c06de90 legacy=0x7f086c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.530+0000 7f088ef92640 10 monclient: shutdown 2024-03-29T11:23:24.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.530+0000 7f088ef92640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:24.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.530+0000 7f088ef92640 1 -- 172.21.15.59:0/2994758201 >> v1:172.21.15.59:6790/0 conn(0x7f08880b4820 legacy=0x7f08881e5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.531+0000 7f088ef92640 1 -- 172.21.15.59:0/2994758201 shutdown_connections 2024-03-29T11:23:24.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.531+0000 7f088ef92640 1 -- 172.21.15.59:0/2994758201 >> 172.21.15.59:0/2994758201 conn(0x7f088806d070 msgr2=0x7f08880b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:24.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.531+0000 7f088ef92640 1 -- 172.21.15.59:0/2994758201 shutdown_connections 2024-03-29T11:23:24.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.532+0000 7f088ef92640 1 -- 172.21.15.59:0/2994758201 wait complete. 2024-03-29T11:23:24.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:24.533+0000 7f088ef92640 1 librados: shutdown 2024-03-29T11:23:24.533 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:31.340 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:31.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.501+0000 7f8de0f4d640 10 monclient: get_monmap_and_config 2024-03-29T11:23:31.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.501+0000 7f8de0f4d640 10 monclient: build_initial_monmap 2024-03-29T11:23:31.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.501+0000 7f8de0f4d640 10 monclient: monmap: 2024-03-29T11:23:31.482 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:31.482 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:31.482 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:23:31.502259+0000 2024-03-29T11:23:31.482 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:23:31.502259+0000 2024-03-29T11:23:31.482 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:31.482 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:31.482 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:31.482 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:23:31.483 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:23:31.483 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:31.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.503+0000 7f8de0f4d640 1 Processor -- start 2024-03-29T11:23:31.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.503+0000 7f8de0f4d640 1 -- start start 2024-03-29T11:23:31.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.503+0000 7f8de0f4d640 10 monclient: init 2024-03-29T11:23:31.484 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.504+0000 7f8de0f4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:31.484 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.504+0000 7f8de0f4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:31.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.504+0000 7f8de0f4d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ddc0a9930 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:31.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.504+0000 7f8de0f4d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ddc149dc0 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.504+0000 7f8de0f4d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ddc14ff40 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.504+0000 7f8de0f4d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc0aae60 con 0x7f8ddc0a9930 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.504+0000 7f8de0f4d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc0b4310 con 0x7f8ddc149dc0 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.504+0000 7f8de0f4d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc0afdd0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.504+0000 7f8de0f4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.504+0000 7f8de0f4d640 10 monclient(hunting): authenticate will time out at 1061.618053s 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.507+0000 7f8dd9d9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8ddc149dc0 0x7f8ddc14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:45664/0 (socket says 172.21.15.59:45664) 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.507+0000 7f8dd9d9b640 1 -- 172.21.15.59:0/2921280355 learned_addr learned my addr 172.21.15.59:0/2921280355 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.508+0000 7f8dd959a640 1 -- 172.21.15.59:0/2921280355 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820992061 0 0) 0x7f8ddc0b4310 con 0x7f8ddc149dc0 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.508+0000 7f8dd959a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.508+0000 7f8dd959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.508+0000 7f8dd959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.509+0000 7f8dd959a640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:23:31.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.509+0000 7f8dd959a640 1 -- 172.21.15.59:0/2921280355 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dc4003520 con 0x7f8ddc149dc0 2024-03-29T11:23:31.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.509+0000 7f8dd959a640 1 -- 172.21.15.59:0/2921280355 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824441175 0 0) 0x7f8ddc0afdd0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.509+0000 7f8dd959a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.509+0000 7f8dd959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.509+0000 7f8dd959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.509+0000 7f8dd959a640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.509+0000 7f8dd959a640 1 -- 172.21.15.59:0/2921280355 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ddc0b4310 con 0x7f8ddc14ff40 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.509+0000 7f8dd959a640 1 -- 172.21.15.59:0/2921280355 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3256678887 0 0) 0x7f8ddc0b4310 con 0x7f8ddc14ff40 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.509+0000 7f8dd959a640 1 -- 172.21.15.59:0/2921280355 >> v1:172.21.15.172:6789/0 conn(0x7f8ddc149dc0 legacy=0x7f8ddc14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 1 -- 172.21.15.59:0/2921280355 >> v1:172.21.15.59:6789/0 conn(0x7f8ddc0a9930 legacy=0x7f8ddc143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 10 monclient: _finish_hunting 0 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 1 monclient: found mon.noname-b 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:23:31.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 1 -- 172.21.15.59:0/2921280355 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc14a950 con 0x7f8ddc14ff40 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 10 monclient: _finish_auth 0 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 10 monclient: _check_auth_tickets 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8de0f4d640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8de0f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 1 -- 172.21.15.59:0/2921280355 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8dd40019f0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8dd959a640 10 monclient: dump: 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:31.494 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.510+0000 7f8de0f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.511+0000 7f8dd959a640 1 -- 172.21.15.59:0/2921280355 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dd4001c10 con 0x7f8ddc14ff40 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.511+0000 7f8dd959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.511+0000 7f8dd959a640 1 -- 172.21.15.59:0/2921280355 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8dd4001fa0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.511+0000 7f8dd959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.511+0000 7f8dd959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.511+0000 7f8dd959a640 10 monclient: dump: 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:31.495 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.511+0000 7f8de0f4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.511+0000 7f8de0f4d640 4 set_mon_vals no callback set 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.511+0000 7f8de0f4d640 10 monclient: shutdown 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.511+0000 7f8de0f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.511+0000 7f8de0f4d640 1 -- 172.21.15.59:0/2921280355 >> v1:172.21.15.59:6790/0 conn(0x7f8ddc14ff40 legacy=0x7f8ddc152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.512+0000 7f8de0f4d640 1 -- 172.21.15.59:0/2921280355 shutdown_connections 2024-03-29T11:23:31.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.512+0000 7f8de0f4d640 1 -- 172.21.15.59:0/2921280355 >> 172.21.15.59:0/2921280355 conn(0x7f8ddc0b35e0 msgr2=0x7f8ddc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:31.497 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.513+0000 7f8de0f4d640 1 -- 172.21.15.59:0/2921280355 shutdown_connections 2024-03-29T11:23:31.497 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.513+0000 7f8de0f4d640 1 -- 172.21.15.59:0/2921280355 wait complete. 2024-03-29T11:23:31.498 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.514+0000 7f8de0f4d640 10 monclient: build_initial_monmap 2024-03-29T11:23:31.498 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.514+0000 7f8de0f4d640 10 monclient: monmap: 2024-03-29T11:23:31.498 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:31.498 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:31.498 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:23:31.498 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:23:31.498 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:31.498 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.515+0000 7f8de0f4d640 1 librados: starting msgr at 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.515+0000 7f8de0f4d640 1 librados: starting objecter 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.515+0000 7f8de0f4d640 1 Processor -- start 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.515+0000 7f8de0f4d640 1 -- start start 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.515+0000 7f8de0f4d640 1 librados: setting wanted keys 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.515+0000 7f8de0f4d640 1 librados: calling monclient init 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.515+0000 7f8de0f4d640 10 monclient: init 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.516+0000 7f8de0f4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.516+0000 7f8de0f4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.516+0000 7f8de0f4d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ddc0a9930 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:31.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.516+0000 7f8de0f4d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ddc149dc0 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:31.500 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.516+0000 7f8de0f4d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ddc14ff40 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:31.500 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.516+0000 7f8de0f4d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc1dbbd0 con 0x7f8ddc0a9930 2024-03-29T11:23:31.500 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.516+0000 7f8de0f4d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc144fe0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.500 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.516+0000 7f8de0f4d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc1dc770 con 0x7f8ddc149dc0 2024-03-29T11:23:31.500 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.516+0000 7f8de0f4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:31.500 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.516+0000 7f8de0f4d640 10 monclient(hunting): authenticate will time out at 1061.629910s 2024-03-29T11:23:31.500 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.518+0000 7f8ddad9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8ddc14ff40 0x7f8ddc1e6400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:48916/0 (socket says 172.21.15.59:48916) 2024-03-29T11:23:31.500 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.518+0000 7f8ddad9d640 1 -- 172.21.15.59:0/1181411294 learned_addr learned my addr 172.21.15.59:0/1181411294 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:31.500 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.521+0000 7f8dcaffd640 10 client.?.objecter ms_handle_connect 0x7f8ddc14ff40 2024-03-29T11:23:31.501 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.521+0000 7f8dcaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:31.501 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.521+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113032602 0 0) 0x7f8ddc144fe0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.501 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.521+0000 7f8dcaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.501 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.521+0000 7f8dcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.501 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.521+0000 7f8dcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.501 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.521+0000 7f8dcaffd640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:23:31.501 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.521+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dbc004eb0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.501 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.522+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2773817620 0 0) 0x7f8dbc004eb0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.501 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.522+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ddc144fe0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8dd4001f60 con 0x7f8ddc14ff40 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 monclient(hunting): dump: 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:31.502 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:31.503 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:31.503 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 client.?.objecter ms_handle_connect 0x7f8ddc149dc0 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2057861730 0 0) 0x7f8ddc144fe0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 >> v1:172.21.15.59:6790/0 conn(0x7f8ddc149dc0 legacy=0x7f8ddc0ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 >> v1:172.21.15.59:6789/0 conn(0x7f8ddc0a9930 legacy=0x7f8ddc1dc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 1 monclient: found mon.b 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc1dc900 con 0x7f8ddc14ff40 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:31.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8de0f4d640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8de0f4d640 10 monclient: _renew_subs 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8de0f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8de0f4d640 1 -- 172.21.15.59:0/1181411294 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc1479f0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8de0f4d640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8de0f4d640 10 monclient: _renew_subs 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8de0f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8de0f4d640 1 -- 172.21.15.59:0/1181411294 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc0aca90 con 0x7f8ddc14ff40 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8de0f4d640 1 librados: init done 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dd40020c0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8dd4003e80 con 0x7f8ddc14ff40 2024-03-29T11:23:31.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.523+0000 7f8dcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.524+0000 7f8dcaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.524+0000 7f8dcaffd640 10 monclient: dump: 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.524+0000 7f8dd8d99640 4 set_mon_vals no callback set 2024-03-29T11:23:31.508 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.526+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f8dd4003fe0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.527+0000 7f8de0f4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:31.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.527+0000 7f8de0f4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:31.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.527+0000 7f8de0f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:31.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.527+0000 7f8de0f4d640 1 -- 172.21.15.59:0/1181411294 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc0005080 con 0x7f8ddc14ff40 2024-03-29T11:23:31.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.527+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f8dd4084ee0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.527+0000 7f8dcaffd640 10 client.4354.objecter ms_dispatch 0x7f8ddc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:31.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.527+0000 7f8dcaffd640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:31.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.527+0000 7f8dcaffd640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:31.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.527+0000 7f8dcaffd640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-29T11:23:31.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.528+0000 7f8dda59c640 10 monclient: get_auth_request con 0x7f8dbc06dc90 auth_method 0 2024-03-29T11:23:31.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.529+0000 7f8dcaffd640 10 client.4354.objecter ms_handle_connect 0x7f8dbc06dc90 2024-03-29T11:23:31.514 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.535+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8dd40505f0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.514 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.535+0000 7f8dcaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:31.514 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.535+0000 7f8dcaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:31.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.875+0000 7f8de0f4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:31.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.875+0000 7f8de0f4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:31.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.875+0000 7f8de0f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:31.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.875+0000 7f8de0f4d640 1 -- 172.21.15.59:0/1181411294 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dc0005330 con 0x7f8ddc14ff40 2024-03-29T11:23:31.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.877+0000 7f8dcaffd640 1 -- 172.21.15.59:0/1181411294 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f8dd40526f0 con 0x7f8ddc14ff40 2024-03-29T11:23:31.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.877+0000 7f8dcaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:31.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.877+0000 7f8dcaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:31.856 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:23:31.857 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:31.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.880+0000 7f8de0f4d640 10 librados: watch_flush enter 2024-03-29T11:23:31.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.880+0000 7f8de0f4d640 10 librados: watch_flush exit 2024-03-29T11:23:31.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.880+0000 7f8de0f4d640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-29T11:23:31.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.880+0000 7f8de0f4d640 10 client.4354.objecter successfully canceled tick 2024-03-29T11:23:31.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.882+0000 7f8de0f4d640 1 -- 172.21.15.59:0/1181411294 >> v1:172.21.15.172:6816/2420742773 conn(0x7f8dbc06dc90 legacy=0x7f8dbc070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.882+0000 7f8de0f4d640 10 monclient: shutdown 2024-03-29T11:23:31.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.882+0000 7f8de0f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:31.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.882+0000 7f8de0f4d640 1 -- 172.21.15.59:0/1181411294 >> v1:172.21.15.172:6789/0 conn(0x7f8ddc14ff40 legacy=0x7f8ddc1e6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.882+0000 7f8de0f4d640 1 -- 172.21.15.59:0/1181411294 shutdown_connections 2024-03-29T11:23:31.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.882+0000 7f8de0f4d640 1 -- 172.21.15.59:0/1181411294 >> 172.21.15.59:0/1181411294 conn(0x7f8ddc0b35e0 msgr2=0x7f8ddc143ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:31.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.884+0000 7f8de0f4d640 1 -- 172.21.15.59:0/1181411294 shutdown_connections 2024-03-29T11:23:31.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.884+0000 7f8de0f4d640 1 -- 172.21.15.59:0/1181411294 wait complete. 2024-03-29T11:23:31.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:31.884+0000 7f8de0f4d640 1 librados: shutdown 2024-03-29T11:23:31.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:38.687 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:38.843 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.863+0000 7f4e0037b640 10 monclient: get_monmap_and_config 2024-03-29T11:23:38.843 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.863+0000 7f4e0037b640 10 monclient: build_initial_monmap 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.863+0000 7f4e0037b640 10 monclient: monmap: 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:23:38.863950+0000 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:23:38.863950+0000 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:38.844 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.864+0000 7f4e0037b640 1 Processor -- start 2024-03-29T11:23:38.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.867+0000 7f4e0037b640 1 -- start start 2024-03-29T11:23:38.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.867+0000 7f4e0037b640 10 monclient: init 2024-03-29T11:23:38.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.868+0000 7f4e0037b640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:38.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.868+0000 7f4e0037b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:38.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.868+0000 7f4e0037b640 10 monclient(hunting): picked mon.noname-b con 0x7f4df80a9ac0 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:38.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.868+0000 7f4e0037b640 10 monclient(hunting): picked mon.noname-a con 0x7f4df813ee90 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:38.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.868+0000 7f4e0037b640 10 monclient(hunting): picked mon.noname-c con 0x7f4df80af900 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:38.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.868+0000 7f4e0037b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80afd10 con 0x7f4df813ee90 2024-03-29T11:23:38.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.868+0000 7f4e0037b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80af6f0 con 0x7f4df80af900 2024-03-29T11:23:38.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.868+0000 7f4e0037b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80aac80 con 0x7f4df80a9ac0 2024-03-29T11:23:38.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.868+0000 7f4e0037b640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:38.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.868+0000 7f4e0037b640 10 monclient(hunting): authenticate will time out at 1068.982388s 2024-03-29T11:23:38.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.872+0000 7f4dfd916640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4df813ee90 0x7f4df813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:45309/0 (socket says 172.21.15.59:45309) 2024-03-29T11:23:38.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.872+0000 7f4dfd916640 1 -- 172.21.15.59:0/2606805929 learned_addr learned my addr 172.21.15.59:0/2606805929 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:38.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 1 -- 172.21.15.59:0/2606805929 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996960305 0 0) 0x7f4df80afd10 con 0x7f4df813ee90 2024-03-29T11:23:38.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 10 monclient(hunting): _negotiate 2024-03-29T11:23:38.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:38.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 1 -- 172.21.15.59:0/2606805929 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de4005730 con 0x7f4df813ee90 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 1 -- 172.21.15.59:0/2606805929 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452891651 0 0) 0x7f4df80aac80 con 0x7f4df80a9ac0 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 10 monclient(hunting): _negotiate 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 1 -- 172.21.15.59:0/2606805929 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df80afd10 con 0x7f4df80a9ac0 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 1 -- 172.21.15.59:0/2606805929 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1253671149 0 0) 0x7f4df80afd10 con 0x7f4df80a9ac0 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 1 -- 172.21.15.59:0/2606805929 >> v1:172.21.15.172:6789/0 conn(0x7f4df80af900 legacy=0x7f4df8149b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 1 -- 172.21.15.59:0/2606805929 >> v1:172.21.15.59:6789/0 conn(0x7f4df813ee90 legacy=0x7f4df813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 10 monclient: _finish_hunting 0 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 1 monclient: found mon.noname-b 2024-03-29T11:23:38.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.874+0000 7f4dfd115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4dfd115640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4dfd115640 1 -- 172.21.15.59:0/2606805929 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df80aaf60 con 0x7f4df80a9ac0 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4dfd115640 10 monclient: _finish_auth 0 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4dfd115640 10 monclient: _check_auth_tickets 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4dfd115640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4e0037b640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4e0037b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4dfd115640 1 -- 172.21.15.59:0/2606805929 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4df40015f0 con 0x7f4df80a9ac0 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4dfd115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4dfd115640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4dfd115640 10 monclient: dump: 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:38.858 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4e0037b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4dfd115640 1 -- 172.21.15.59:0/2606805929 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4df4001810 con 0x7f4df80a9ac0 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4dfd115640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4e0037b640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.875+0000 7f4e0037b640 4 set_mon_vals no callback set 2024-03-29T11:23:38.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.876+0000 7f4e0037b640 10 monclient: shutdown 2024-03-29T11:23:38.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.876+0000 7f4e0037b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:38.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.876+0000 7f4e0037b640 1 -- 172.21.15.59:0/2606805929 >> v1:172.21.15.59:6790/0 conn(0x7f4df80a9ac0 legacy=0x7f4df80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:38.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.878+0000 7f4e0037b640 1 -- 172.21.15.59:0/2606805929 shutdown_connections 2024-03-29T11:23:38.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.878+0000 7f4e0037b640 1 -- 172.21.15.59:0/2606805929 >> 172.21.15.59:0/2606805929 conn(0x7f4df80b35e0 msgr2=0x7f4df80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:38.860 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.881+0000 7f4e0037b640 1 -- 172.21.15.59:0/2606805929 shutdown_connections 2024-03-29T11:23:38.862 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.881+0000 7f4e0037b640 1 -- 172.21.15.59:0/2606805929 wait complete. 2024-03-29T11:23:38.862 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.882+0000 7f4e0037b640 10 monclient: build_initial_monmap 2024-03-29T11:23:38.862 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.882+0000 7f4e0037b640 10 monclient: monmap: 2024-03-29T11:23:38.862 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:38.862 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:38.862 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:23:38.862 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:23:38.862 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:38.862 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:38.862 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:38.862 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:23:38.863 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:23:38.863 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:38.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.882+0000 7f4e0037b640 1 librados: starting msgr at 2024-03-29T11:23:38.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.882+0000 7f4e0037b640 1 librados: starting objecter 2024-03-29T11:23:38.863 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.882+0000 7f4e0037b640 1 Processor -- start 2024-03-29T11:23:38.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.883+0000 7f4e0037b640 1 -- start start 2024-03-29T11:23:38.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.883+0000 7f4e0037b640 1 librados: setting wanted keys 2024-03-29T11:23:38.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.883+0000 7f4e0037b640 1 librados: calling monclient init 2024-03-29T11:23:38.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.883+0000 7f4e0037b640 10 monclient: init 2024-03-29T11:23:38.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.883+0000 7f4e0037b640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:38.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.883+0000 7f4e0037b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:38.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.883+0000 7f4e0037b640 10 monclient(hunting): picked mon.noname-b con 0x7f4df80a9ac0 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:38.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.883+0000 7f4e0037b640 10 monclient(hunting): picked mon.noname-c con 0x7f4df80af900 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:38.866 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.887+0000 7f4e0037b640 10 monclient(hunting): picked mon.noname-a con 0x7f4df813ee90 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:38.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.887+0000 7f4e0037b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80b2ec0 con 0x7f4df813ee90 2024-03-29T11:23:38.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.887+0000 7f4e0037b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80b29a0 con 0x7f4df80a9ac0 2024-03-29T11:23:38.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.887+0000 7f4e0037b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80b2b30 con 0x7f4df80af900 2024-03-29T11:23:38.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.887+0000 7f4e0037b640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:38.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.887+0000 7f4e0037b640 10 monclient(hunting): authenticate will time out at 1069.000937s 2024-03-29T11:23:38.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.891+0000 7f4dfe918640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4df813ee90 0x7f4df81ee840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:45439/0 (socket says 172.21.15.59:45439) 2024-03-29T11:23:38.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.891+0000 7f4dfe918640 1 -- 172.21.15.59:0/3372213700 learned_addr learned my addr 172.21.15.59:0/3372213700 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:38.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.891+0000 7f4deeffd640 10 client.?.objecter ms_handle_connect 0x7f4df813ee90 2024-03-29T11:23:38.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.891+0000 7f4deeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:38.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.891+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185393393 0 0) 0x7f4df80b2ec0 con 0x7f4df813ee90 2024-03-29T11:23:38.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.891+0000 7f4deeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:38.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.891+0000 7f4deeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:38.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.891+0000 7f4deeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:38.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.891+0000 7f4deeffd640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:23:38.874 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.891+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ddc004eb0 con 0x7f4df813ee90 2024-03-29T11:23:38.874 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.894+0000 7f4deeffd640 10 client.?.objecter ms_handle_connect 0x7f4df80a9ac0 2024-03-29T11:23:38.874 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.894+0000 7f4deeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.894+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101847899 0 0) 0x7f4df80b29a0 con 0x7f4df80a9ac0 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.894+0000 7f4deeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.894+0000 7f4deeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.894+0000 7f4deeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.894+0000 7f4deeffd640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.894+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df80b2ec0 con 0x7f4df80a9ac0 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2658620053 0 0) 0x7f4df80b2ec0 con 0x7f4df80a9ac0 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4df80b29a0 con 0x7f4df80a9ac0 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 842811255 0 0) 0x7f4ddc004eb0 con 0x7f4df813ee90 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4df80b2ec0 con 0x7f4df813ee90 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4df4003790 con 0x7f4df80a9ac0 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:38.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 10 monclient(hunting): dump: 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4de8001770 con 0x7f4df813ee90 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:38.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.896+0000 7f4deeffd640 10 monclient(hunting): dump: 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 856501867 0 0) 0x7f4df80b2ec0 con 0x7f4df813ee90 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 >> v1:172.21.15.59:6790/0 conn(0x7f4df80af900 legacy=0x7f4df81e7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 >> v1:172.21.15.172:6789/0 conn(0x7f4df80a9ac0 legacy=0x7f4df80b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:38.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 1 monclient: found mon.a 2024-03-29T11:23:38.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:38.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:38.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df813d480 con 0x7f4df813ee90 2024-03-29T11:23:38.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:38.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:38.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:38.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 10 client.?.objecter ms_handle_connect 0x7f4df80af900 2024-03-29T11:23:38.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.897+0000 7f4deeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:38.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.898+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4de8001c80 con 0x7f4df813ee90 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.898+0000 7f4deeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.898+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4de8004aa0 con 0x7f4df813ee90 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.898+0000 7f4deeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.898+0000 7f4deeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.898+0000 7f4deeffd640 10 monclient: dump: 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:38.882 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.898+0000 7f4dfc914640 4 set_mon_vals no callback set 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.899+0000 7f4e0037b640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.899+0000 7f4e0037b640 10 monclient: _renew_subs 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.899+0000 7f4e0037b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.899+0000 7f4e0037b640 1 -- 172.21.15.59:0/3372213700 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df813d6d0 con 0x7f4df813ee90 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.899+0000 7f4e0037b640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.899+0000 7f4e0037b640 10 monclient: _renew_subs 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.899+0000 7f4e0037b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.899+0000 7f4e0037b640 1 -- 172.21.15.59:0/3372213700 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df81e7bf0 con 0x7f4df813ee90 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.899+0000 7f4e0037b640 1 librados: init done 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.901+0000 7f4e0037b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.901+0000 7f4e0037b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.901+0000 7f4e0037b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:38.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.901+0000 7f4e0037b640 1 -- 172.21.15.59:0/3372213700 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df80b2cc0 con 0x7f4df813ee90 2024-03-29T11:23:38.884 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.904+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f4de801aa70 con 0x7f4df813ee90 2024-03-29T11:23:38.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.905+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f4de8084a10 con 0x7f4df813ee90 2024-03-29T11:23:38.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.905+0000 7f4deeffd640 10 client.4377.objecter ms_dispatch 0x7f4df8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:38.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.905+0000 7f4deeffd640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:38.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.905+0000 7f4deeffd640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:38.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.905+0000 7f4deeffd640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-29T11:23:38.888 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.909+0000 7f4dfe117640 10 monclient: get_auth_request con 0x7f4ddc06de00 auth_method 0 2024-03-29T11:23:38.889 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.910+0000 7f4deeffd640 10 client.4377.objecter ms_handle_connect 0x7f4ddc06de00 2024-03-29T11:23:38.892 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.913+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4df80b2cc0 con 0x7f4df813ee90 2024-03-29T11:23:38.892 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.913+0000 7f4deeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:38.892 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:38.913+0000 7f4deeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:39.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.241+0000 7f4e0037b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.241+0000 7f4e0037b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.241+0000 7f4e0037b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:39.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.241+0000 7f4e0037b640 1 -- 172.21.15.59:0/3372213700 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4df80b2cc0 con 0x7f4df813ee90 2024-03-29T11:23:39.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.245+0000 7f4deeffd640 1 -- 172.21.15.59:0/3372213700 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f4df80b2cc0 con 0x7f4df813ee90 2024-03-29T11:23:39.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.245+0000 7f4deeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.245+0000 7f4deeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:39.227 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:23:39.227 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:39.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.249+0000 7f4e0037b640 10 librados: watch_flush enter 2024-03-29T11:23:39.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.249+0000 7f4e0037b640 10 librados: watch_flush exit 2024-03-29T11:23:39.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.249+0000 7f4e0037b640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-29T11:23:39.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.249+0000 7f4e0037b640 10 client.4377.objecter successfully canceled tick 2024-03-29T11:23:39.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.250+0000 7f4e0037b640 1 -- 172.21.15.59:0/3372213700 >> v1:172.21.15.172:6816/2420742773 conn(0x7f4ddc06de00 legacy=0x7f4ddc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.250+0000 7f4e0037b640 10 monclient: shutdown 2024-03-29T11:23:39.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.250+0000 7f4e0037b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:39.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.250+0000 7f4e0037b640 1 -- 172.21.15.59:0/3372213700 >> v1:172.21.15.59:6789/0 conn(0x7f4df813ee90 legacy=0x7f4df81ee840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.253+0000 7f4e0037b640 1 -- 172.21.15.59:0/3372213700 shutdown_connections 2024-03-29T11:23:39.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.253+0000 7f4e0037b640 1 -- 172.21.15.59:0/3372213700 >> 172.21.15.59:0/3372213700 conn(0x7f4df80b35e0 msgr2=0x7f4df813a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:39.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.253+0000 7f4e0037b640 1 -- 172.21.15.59:0/3372213700 shutdown_connections 2024-03-29T11:23:39.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.253+0000 7f4e0037b640 1 -- 172.21.15.59:0/3372213700 wait complete. 2024-03-29T11:23:39.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:39.253+0000 7f4e0037b640 1 librados: shutdown 2024-03-29T11:23:39.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:46.057 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:46.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.224+0000 7fe5bd0d0640 10 monclient: get_monmap_and_config 2024-03-29T11:23:46.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.224+0000 7fe5bd0d0640 10 monclient: build_initial_monmap 2024-03-29T11:23:46.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.224+0000 7fe5bd0d0640 10 monclient: monmap: 2024-03-29T11:23:46.204 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:46.204 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:46.204 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:23:46.225508+0000 2024-03-29T11:23:46.205 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:23:46.225508+0000 2024-03-29T11:23:46.205 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:46.205 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:46.205 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:46.205 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:23:46.205 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:23:46.205 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:46.206 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.225+0000 7fe5bd0d0640 1 Processor -- start 2024-03-29T11:23:46.206 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.226+0000 7fe5bd0d0640 1 -- start start 2024-03-29T11:23:46.206 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.226+0000 7fe5bd0d0640 10 monclient: init 2024-03-29T11:23:46.206 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.226+0000 7fe5bd0d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:46.206 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.226+0000 7fe5bd0d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:46.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.229+0000 7fe5bd0d0640 10 monclient(hunting): picked mon.noname-b con 0x7fe5b0037230 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:46.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.229+0000 7fe5b7fff640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fe5b0037230 0x7fe5b0036080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:54789/0 (socket says 172.21.15.59:54789) 2024-03-29T11:23:46.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.230+0000 7fe5bd0d0640 10 monclient(hunting): picked mon.noname-c con 0x7fe5b0033a80 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:46.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.230+0000 7fe5b7fff640 1 -- 172.21.15.59:0/61653422 learned_addr learned my addr 172.21.15.59:0/61653422 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:46.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.230+0000 7fe5bd0d0640 10 monclient(hunting): picked mon.noname-a con 0x7fe5b00d4770 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:46.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.230+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/61653422 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b0035bf0 con 0x7fe5b00d4770 2024-03-29T11:23:46.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.230+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/61653422 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b002c150 con 0x7fe5b0033a80 2024-03-29T11:23:46.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.230+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/61653422 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b00356f0 con 0x7fe5b0037230 2024-03-29T11:23:46.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.230+0000 7fe5bd0d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:46.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.230+0000 7fe5bd0d0640 10 monclient(hunting): authenticate will time out at 1076.343877s 2024-03-29T11:23:46.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 1 -- 172.21.15.59:0/61653422 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690121485 0 0) 0x7fe5b00356f0 con 0x7fe5b0037230 2024-03-29T11:23:46.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:46.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:46.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:46.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:23:46.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 1 -- 172.21.15.59:0/61653422 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a8003520 con 0x7fe5b0037230 2024-03-29T11:23:46.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 1 -- 172.21.15.59:0/61653422 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1853407966 0 0) 0x7fe5a8003520 con 0x7fe5b0037230 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 1 -- 172.21.15.59:0/61653422 >> v1:172.21.15.172:6789/0 conn(0x7fe5b0033a80 legacy=0x7fe5b00cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 1 -- 172.21.15.59:0/61653422 >> v1:172.21.15.59:6789/0 conn(0x7fe5b00d4770 legacy=0x7fe5b00d6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 1 -- 172.21.15.59:0/61653422 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b0035880 con 0x7fe5b0037230 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 1 -- 172.21.15.59:0/61653422 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe5ac001690 con 0x7fe5b0037230 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient: dump: 2024-03-29T11:23:46.215 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5bd0d0640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5bd0d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 1 -- 172.21.15.59:0/61653422 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5ac001860 con 0x7fe5b0037230 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:46.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 1 -- 172.21.15.59:0/61653422 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe5ac001cb0 con 0x7fe5b0037230 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.233+0000 7fe5b6ffd640 10 monclient: dump: 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.234+0000 7fe5bd0d0640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:46.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.234+0000 7fe5bd0d0640 4 set_mon_vals no callback set 2024-03-29T11:23:46.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.234+0000 7fe5bd0d0640 10 monclient: shutdown 2024-03-29T11:23:46.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.234+0000 7fe5bd0d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:46.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.234+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/61653422 >> v1:172.21.15.59:6790/0 conn(0x7fe5b0037230 legacy=0x7fe5b0036080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.235+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/61653422 shutdown_connections 2024-03-29T11:23:46.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.235+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/61653422 >> 172.21.15.59:0/61653422 conn(0x7fe5b003a590 msgr2=0x7fe5b00367e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:46.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.238+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/61653422 shutdown_connections 2024-03-29T11:23:46.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.238+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/61653422 wait complete. 2024-03-29T11:23:46.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.239+0000 7fe5bd0d0640 10 monclient: build_initial_monmap 2024-03-29T11:23:46.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.239+0000 7fe5bd0d0640 10 monclient: monmap: 2024-03-29T11:23:46.223 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:46.223 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.240+0000 7fe5bd0d0640 1 librados: starting msgr at 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.240+0000 7fe5bd0d0640 1 librados: starting objecter 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.240+0000 7fe5bd0d0640 1 Processor -- start 2024-03-29T11:23:46.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.241+0000 7fe5bd0d0640 1 -- start start 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.241+0000 7fe5bd0d0640 1 librados: setting wanted keys 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.241+0000 7fe5bd0d0640 1 librados: calling monclient init 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.241+0000 7fe5bd0d0640 10 monclient: init 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.242+0000 7fe5bd0d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.242+0000 7fe5bd0d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.244+0000 7fe5bd0d0640 10 monclient(hunting): picked mon.noname-a con 0x7fe5b0033a80 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.244+0000 7fe5bd0d0640 10 monclient(hunting): picked mon.noname-b con 0x7fe5b00d4770 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.244+0000 7fe5bd0d0640 10 monclient(hunting): picked mon.noname-c con 0x7fe5b00cbea0 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.244+0000 7fe5bd0d0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b00cc3c0 con 0x7fe5b0033a80 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.244+0000 7fe5bd0d0640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b016c6e0 con 0x7fe5b00d4770 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.244+0000 7fe5bd0d0640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b00c5a40 con 0x7fe5b00cbea0 2024-03-29T11:23:46.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.244+0000 7fe5bd0d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:46.226 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.244+0000 7fe5bd0d0640 10 monclient(hunting): authenticate will time out at 1076.357798s 2024-03-29T11:23:46.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.248+0000 7fe5b7fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe5b0033a80 0x7fe5b00c9a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39263/0 (socket says 172.21.15.59:39263) 2024-03-29T11:23:46.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.248+0000 7fe5b7fff640 1 -- 172.21.15.59:0/560717478 learned_addr learned my addr 172.21.15.59:0/560717478 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:46.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.248+0000 7fe5b4ff9640 10 client.?.objecter ms_handle_connect 0x7fe5b0033a80 2024-03-29T11:23:46.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.248+0000 7fe5b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:46.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.249+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010299471 0 0) 0x7fe5b00cc3c0 con 0x7fe5b0033a80 2024-03-29T11:23:46.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.249+0000 7fe5b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:46.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.249+0000 7fe5b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:46.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.249+0000 7fe5b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:46.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.249+0000 7fe5b4ff9640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:23:46.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.249+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe59c004eb0 con 0x7fe5b0033a80 2024-03-29T11:23:46.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.249+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793244957 0 0) 0x7fe59c004eb0 con 0x7fe5b0033a80 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.249+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5b00cc3c0 con 0x7fe5b0033a80 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.249+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe5ac001b70 con 0x7fe5b0033a80 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.249+0000 7fe5b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.250+0000 7fe5b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.250+0000 7fe5b4ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:46.231 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:46.232 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:46.232 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:46.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.251+0000 7fe5b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:46.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.251+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4169628855 0 0) 0x7fe5b00cc3c0 con 0x7fe5b0033a80 2024-03-29T11:23:46.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.251+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 >> v1:172.21.15.59:6790/0 conn(0x7fe5b00cbea0 legacy=0x7fe5b016bfd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:46.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.251+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 >> v1:172.21.15.172:6789/0 conn(0x7fe5b00d4770 legacy=0x7fe5b016b8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.253+0000 7fe5b4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.253+0000 7fe5b4ff9640 1 monclient: found mon.a 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.253+0000 7fe5b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.253+0000 7fe5b4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.253+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b00cc850 con 0x7fe5b0033a80 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.253+0000 7fe5b4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.253+0000 7fe5b4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.253+0000 7fe5b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.254+0000 7fe5bd0d0640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.254+0000 7fe5bd0d0640 10 monclient: _renew_subs 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.254+0000 7fe5bd0d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.254+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/560717478 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b00d0a10 con 0x7fe5b0033a80 2024-03-29T11:23:46.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.254+0000 7fe5bd0d0640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:46.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.254+0000 7fe5bd0d0640 10 monclient: _renew_subs 2024-03-29T11:23:46.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.254+0000 7fe5bd0d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:46.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.254+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/560717478 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b00ca120 con 0x7fe5b0033a80 2024-03-29T11:23:46.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.254+0000 7fe5bd0d0640 1 librados: init done 2024-03-29T11:23:46.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.255+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5ac004bf0 con 0x7fe5b0033a80 2024-03-29T11:23:46.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.255+0000 7fe5b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:46.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.255+0000 7fe5b67fc640 4 set_mon_vals no callback set 2024-03-29T11:23:46.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.255+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fe5ac004f00 con 0x7fe5b0033a80 2024-03-29T11:23:46.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.256+0000 7fe5b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.256+0000 7fe5b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:46.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.256+0000 7fe5b4ff9640 10 monclient: dump: 2024-03-29T11:23:46.236 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:46.236 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:46.237 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.256+0000 7fe5bd0d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.257+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fe5ac01aed0 con 0x7fe5b0033a80 2024-03-29T11:23:46.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.258+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fe5ac084ca0 con 0x7fe5b0033a80 2024-03-29T11:23:46.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.258+0000 7fe5b4ff9640 10 client.4380.objecter ms_dispatch 0x7fe5b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:46.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.258+0000 7fe5b4ff9640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:46.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.258+0000 7fe5b4ff9640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:46.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.258+0000 7fe5b4ff9640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T11:23:46.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.258+0000 7fe5bd0d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.258+0000 7fe5bd0d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:46.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.258+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/560717478 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe598005080 con 0x7fe5b0033a80 2024-03-29T11:23:46.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.258+0000 7fe5b77fe640 10 monclient: get_auth_request con 0x7fe59c06dc00 auth_method 0 2024-03-29T11:23:46.239 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.258+0000 7fe5b4ff9640 10 client.4380.objecter ms_handle_connect 0x7fe59c06dc00 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.266+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5ac050580 con 0x7fe5b0033a80 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.266+0000 7fe5b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.249 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.266+0000 7fe5b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:46.579 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.599+0000 7fe5bd0d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:46.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.599+0000 7fe5bd0d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:46.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.599+0000 7fe5bd0d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:23:46.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.599+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/560717478 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe598005330 con 0x7fe5b0033a80 2024-03-29T11:23:46.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.600+0000 7fe5b4ff9640 1 -- 172.21.15.59:0/560717478 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fe5ac0524b0 con 0x7fe5b0033a80 2024-03-29T11:23:46.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.600+0000 7fe5b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:46.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.600+0000 7fe5b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:46.581 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:23:46.581 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:46.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.605+0000 7fe5bd0d0640 10 librados: watch_flush enter 2024-03-29T11:23:46.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.605+0000 7fe5bd0d0640 10 librados: watch_flush exit 2024-03-29T11:23:46.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.605+0000 7fe5bd0d0640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T11:23:46.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.605+0000 7fe5bd0d0640 10 client.4380.objecter successfully canceled tick 2024-03-29T11:23:46.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.605+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/560717478 >> v1:172.21.15.172:6816/2420742773 conn(0x7fe59c06dc00 legacy=0x7fe59c0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.605+0000 7fe5bd0d0640 10 monclient: shutdown 2024-03-29T11:23:46.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.605+0000 7fe5bd0d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:46.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.605+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/560717478 >> v1:172.21.15.59:6789/0 conn(0x7fe5b0033a80 legacy=0x7fe5b00c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.605+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/560717478 shutdown_connections 2024-03-29T11:23:46.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.605+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/560717478 >> 172.21.15.59:0/560717478 conn(0x7fe5b003a590 msgr2=0x7fe5b00c8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:46.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.606+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/560717478 shutdown_connections 2024-03-29T11:23:46.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.606+0000 7fe5bd0d0640 1 -- 172.21.15.59:0/560717478 wait complete. 2024-03-29T11:23:46.587 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:46.606+0000 7fe5bd0d0640 1 librados: shutdown 2024-03-29T11:23:46.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:53.413 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:53.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.614+0000 7f6da042a640 10 monclient: get_monmap_and_config 2024-03-29T11:23:53.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.614+0000 7f6da042a640 10 monclient: build_initial_monmap 2024-03-29T11:23:53.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.614+0000 7f6da042a640 10 monclient: monmap: 2024-03-29T11:23:53.595 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:53.595 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:53.595 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:23:53.615200+0000 2024-03-29T11:23:53.595 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:23:53.615200+0000 2024-03-29T11:23:53.595 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:53.595 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:53.595 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:53.595 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:23:53.595 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:23:53.595 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:53.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.615+0000 7f6da042a640 1 Processor -- start 2024-03-29T11:23:53.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.615+0000 7f6da042a640 1 -- start start 2024-03-29T11:23:53.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.615+0000 7f6da042a640 10 monclient: init 2024-03-29T11:23:53.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.616+0000 7f6da042a640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:53.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.616+0000 7f6da042a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:53.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.616+0000 7f6da042a640 10 monclient(hunting): picked mon.noname-c con 0x7f6d981422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:53.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.616+0000 7f6da042a640 10 monclient(hunting): picked mon.noname-b con 0x7f6d98142690 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.616+0000 7f6da042a640 10 monclient(hunting): picked mon.noname-a con 0x7f6d980b17d0 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.616+0000 7f6da042a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d98142c80 con 0x7f6d980b17d0 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.616+0000 7f6da042a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d980aae40 con 0x7f6d981422b0 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.616+0000 7f6da042a640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9813fbd0 con 0x7f6d98142690 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.616+0000 7f6da042a640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:53.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.616+0000 7f6da042a640 10 monclient(hunting): authenticate will time out at 1083.730523s 2024-03-29T11:23:53.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.617+0000 7f6d9e1c6640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6d981422b0 0x7f6d9813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:49112/0 (socket says 172.21.15.59:49112) 2024-03-29T11:23:53.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.617+0000 7f6d9e1c6640 1 -- 172.21.15.59:0/364641404 learned_addr learned my addr 172.21.15.59:0/364641404 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:53.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202703240 0 0) 0x7f6d980aae40 con 0x7f6d981422b0 2024-03-29T11:23:53.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:23:53.598 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d84003520 con 0x7f6d981422b0 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022829780 0 0) 0x7f6d98142c80 con 0x7f6d980b17d0 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d980aae40 con 0x7f6d980b17d0 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420909615 0 0) 0x7f6d9813fbd0 con 0x7f6d98142690 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.618+0000 7f6d9d1c4640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d840039d0 con 0x7f6d98142690 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635392361 0 0) 0x7f6d84003520 con 0x7f6d981422b0 2024-03-29T11:23:53.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 >> v1:172.21.15.59:6790/0 conn(0x7f6d98142690 legacy=0x7f6d980b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 >> v1:172.21.15.59:6789/0 conn(0x7f6d980b17d0 legacy=0x7f6d980b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 10 monclient: _finish_hunting 0 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 1 monclient: found mon.noname-c 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d98151940 con 0x7f6d981422b0 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 10 monclient: _finish_auth 0 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 10 monclient: _check_auth_tickets 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f6d88001690 con 0x7f6d981422b0 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6d9d1c4640 10 monclient: dump: 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:53.601 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6da042a640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:23:53.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.619+0000 7f6da042a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:53.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.620+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d880017f0 con 0x7f6d981422b0 2024-03-29T11:23:53.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.620+0000 7f6d9d1c4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:53.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.620+0000 7f6d9d1c4640 1 -- 172.21.15.59:0/364641404 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f6d88001bc0 con 0x7f6d981422b0 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.620+0000 7f6d9d1c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.620+0000 7f6d9d1c4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.620+0000 7f6d9d1c4640 10 monclient: dump: 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.620+0000 7f6da042a640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.620+0000 7f6da042a640 4 set_mon_vals no callback set 2024-03-29T11:23:53.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.621+0000 7f6da042a640 10 monclient: shutdown 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.621+0000 7f6da042a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.621+0000 7f6da042a640 1 -- 172.21.15.59:0/364641404 >> v1:172.21.15.172:6789/0 conn(0x7f6d981422b0 legacy=0x7f6d9813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.621+0000 7f6da042a640 1 -- 172.21.15.59:0/364641404 shutdown_connections 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.621+0000 7f6da042a640 1 -- 172.21.15.59:0/364641404 >> 172.21.15.59:0/364641404 conn(0x7f6d98066a00 msgr2=0x7f6d98066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.621+0000 7f6da042a640 1 -- 172.21.15.59:0/364641404 shutdown_connections 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.622+0000 7f6da042a640 1 -- 172.21.15.59:0/364641404 wait complete. 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.623+0000 7f6da042a640 10 monclient: build_initial_monmap 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.623+0000 7f6da042a640 10 monclient: monmap: 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:53.605 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:53.606 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:23:53.606 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:23:53.606 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:23:53.606 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:53.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.623+0000 7f6da042a640 1 librados: starting msgr at 2024-03-29T11:23:53.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.623+0000 7f6da042a640 1 librados: starting objecter 2024-03-29T11:23:53.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.623+0000 7f6da042a640 1 Processor -- start 2024-03-29T11:23:53.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.623+0000 7f6da042a640 1 -- start start 2024-03-29T11:23:53.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.623+0000 7f6da042a640 1 librados: setting wanted keys 2024-03-29T11:23:53.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.623+0000 7f6da042a640 1 librados: calling monclient init 2024-03-29T11:23:53.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.623+0000 7f6da042a640 10 monclient: init 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6da042a640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6da042a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6da042a640 10 monclient(hunting): picked mon.noname-c con 0x7f6d980b17d0 addr v1:172.21.15.59:6790/0 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6da042a640 10 monclient(hunting): picked mon.noname-b con 0x7f6d981422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6da042a640 10 monclient(hunting): picked mon.noname-a con 0x7f6d98142690 addr v1:172.21.15.59:6789/0 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6da042a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d98131f40 con 0x7f6d98142690 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6da042a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d98132ae0 con 0x7f6d981422b0 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6da042a640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9806eb20 con 0x7f6d980b17d0 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6da042a640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:53.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6da042a640 10 monclient(hunting): authenticate will time out at 1083.738343s 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6d9e9c7640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6d98142690 0x7f6d9812e180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:53614/0 (socket says 172.21.15.59:53614) 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6d9e9c7640 1 -- 172.21.15.59:0/1620537079 learned_addr learned my addr 172.21.15.59:0/1620537079 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.624+0000 7f6d9e1c6640 1 --1- 172.21.15.59:0/1620537079 >> v1:172.21.15.59:6790/0 conn(0x7f6d980b17d0 0x7f6d981323d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:38708/0 (socket says 172.21.15.59:38708) 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 client.?.objecter ms_handle_connect 0x7f6d98142690 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 client.?.objecter ms_handle_connect 0x7f6d980b17d0 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 client.?.objecter ms_handle_connect 0x7f6d981422b0 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915940146 0 0) 0x7f6d98131f40 con 0x7f6d98142690 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:23:53.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7c004eb0 con 0x7f6d98142690 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086441357 0 0) 0x7f6d98132ae0 con 0x7f6d981422b0 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.625+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d98131f40 con 0x7f6d981422b0 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115620706 0 0) 0x7f6d9806eb20 con 0x7f6d980b17d0 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7c005360 con 0x7f6d980b17d0 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2886034289 0 0) 0x7f6d98131f40 con 0x7f6d981422b0 2024-03-29T11:23:53.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d9806eb20 con 0x7f6d981422b0 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f6d90001b90 con 0x7f6d981422b0 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 10 monclient(hunting): dump: 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:53.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.626+0000 7f6d8effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 326943457 0 0) 0x7f6d7c004eb0 con 0x7f6d98142690 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d98131f40 con 0x7f6d98142690 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f6d940019b0 con 0x7f6d98142690 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 10 monclient(hunting): dump: 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:53.613 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1452892265 0 0) 0x7f6d7c005360 con 0x7f6d980b17d0 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d7c004eb0 con 0x7f6d980b17d0 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f6d88001bc0 con 0x7f6d980b17d0 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 10 monclient(hunting): dump: 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:53.614 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 766297640 0 0) 0x7f6d9806eb20 con 0x7f6d981422b0 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 >> v1:172.21.15.59:6790/0 conn(0x7f6d980b17d0 legacy=0x7f6d981323d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 >> v1:172.21.15.59:6789/0 conn(0x7f6d98142690 legacy=0x7f6d9812e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 1 monclient: found mon.b 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9806ee00 con 0x7f6d981422b0 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 10 monclient: _finish_auth 0 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6d8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6da042a640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:23:53.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6da042a640 10 monclient: _renew_subs 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6da042a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6da042a640 1 -- 172.21.15.59:0/1620537079 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9812e890 con 0x7f6d981422b0 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6da042a640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6da042a640 10 monclient: _renew_subs 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6da042a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6da042a640 1 -- 172.21.15.59:0/1620537079 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9813bcf0 con 0x7f6d981422b0 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.627+0000 7f6da042a640 1 librados: init done 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.628+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d90002080 con 0x7f6d981422b0 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.628+0000 7f6d8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.628+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f6d90003b10 con 0x7f6d981422b0 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.628+0000 7f6d8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.628+0000 7f6d9c9c3640 4 set_mon_vals no callback set 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.628+0000 7f6d8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.628+0000 7f6d8effd640 10 monclient: dump: 2024-03-29T11:23:53.616 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:23:53.617 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.629+0000 7f6da042a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.631+0000 7f6da042a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.631+0000 7f6da042a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.631+0000 7f6da042a640 1 -- 172.21.15.59:0/1620537079 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d98075f10 con 0x7f6d981422b0 2024-03-29T11:23:53.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.632+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f6d90019b60 con 0x7f6d981422b0 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.633+0000 7f6d9e1c6640 10 monclient: get_auth_request con 0x7f6d7c06e490 auth_method 0 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.633+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f6d90053730 con 0x7f6d981422b0 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.633+0000 7f6d8effd640 10 client.4363.objecter ms_dispatch 0x7f6d98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.633+0000 7f6d8effd640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.633+0000 7f6d8effd640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.633+0000 7f6d8effd640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-29T11:23:53.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.633+0000 7f6d8effd640 10 client.4363.objecter ms_handle_connect 0x7f6d7c06e490 2024-03-29T11:23:53.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.641+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d9004f9d0 con 0x7f6d981422b0 2024-03-29T11:23:53.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.641+0000 7f6d8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:53.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:53.641+0000 7f6d8effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:54.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.062+0000 7f6da042a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:54.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.063+0000 7f6da042a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:54.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.063+0000 7f6da042a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:23:54.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.063+0000 7f6da042a640 1 -- 172.21.15.59:0/1620537079 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d981320f0 con 0x7f6d981422b0 2024-03-29T11:23:54.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.064+0000 7f6d8effd640 1 -- 172.21.15.59:0/1620537079 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f6d981320f0 con 0x7f6d981422b0 2024-03-29T11:23:54.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.064+0000 7f6d8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:54.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.064+0000 7f6d8effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:54.044 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:23:54.044 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.067+0000 7f6da042a640 10 librados: watch_flush enter 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.067+0000 7f6da042a640 10 librados: watch_flush exit 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.067+0000 7f6da042a640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.067+0000 7f6da042a640 10 client.4363.objecter successfully canceled tick 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.067+0000 7f6da042a640 1 -- 172.21.15.59:0/1620537079 >> v1:172.21.15.172:6816/2420742773 conn(0x7f6d7c06e490 legacy=0x7f6d7c070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.067+0000 7f6da042a640 10 monclient: shutdown 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.067+0000 7f6da042a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.067+0000 7f6da042a640 1 -- 172.21.15.59:0/1620537079 >> v1:172.21.15.172:6789/0 conn(0x7f6d981422b0 legacy=0x7f6d9812da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.068+0000 7f6da042a640 1 -- 172.21.15.59:0/1620537079 shutdown_connections 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.068+0000 7f6da042a640 1 -- 172.21.15.59:0/1620537079 >> 172.21.15.59:0/1620537079 conn(0x7f6d98066a00 msgr2=0x7f6d980b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.068+0000 7f6da042a640 1 -- 172.21.15.59:0/1620537079 shutdown_connections 2024-03-29T11:23:54.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.068+0000 7f6da042a640 1 -- 172.21.15.59:0/1620537079 wait complete. 2024-03-29T11:23:54.049 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:23:54.068+0000 7f6da042a640 1 librados: shutdown 2024-03-29T11:23:54.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:00.870 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:01.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.045+0000 7f8f22ce6640 10 monclient: get_monmap_and_config 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.045+0000 7f8f22ce6640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.045+0000 7f8f22ce6640 10 monclient: monmap: 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:24:01.045975+0000 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:24:01.045975+0000 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:24:01.026 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:01.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.047+0000 7f8f22ce6640 1 Processor -- start 2024-03-29T11:24:01.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.047+0000 7f8f22ce6640 1 -- start start 2024-03-29T11:24:01.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.047+0000 7f8f22ce6640 10 monclient: init 2024-03-29T11:24:01.029 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.048+0000 7f8f22ce6640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.048+0000 7f8f22ce6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:01.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.048+0000 7f8f22ce6640 10 monclient(hunting): picked mon.noname-c con 0x7f8f1c0ae500 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:01.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.048+0000 7f8f22ce6640 10 monclient(hunting): picked mon.noname-b con 0x7f8f1c0add80 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:01.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.048+0000 7f8f22ce6640 10 monclient(hunting): picked mon.noname-a con 0x7f8f1c0ad0a0 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:01.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.048+0000 7f8f22ce6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c0a9f00 con 0x7f8f1c0ad0a0 2024-03-29T11:24:01.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.048+0000 7f8f22ce6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c145eb0 con 0x7f8f1c0ae500 2024-03-29T11:24:01.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.048+0000 7f8f22ce6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c0a94e0 con 0x7f8f1c0add80 2024-03-29T11:24:01.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.048+0000 7f8f22ce6640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.048+0000 7f8f22ce6640 10 monclient(hunting): authenticate will time out at 1091.162145s 2024-03-29T11:24:01.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.053+0000 7f8f20a82640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8f1c0ae500 0x7f8f1c1457a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:33191/0 (socket says 172.21.15.59:33191) 2024-03-29T11:24:01.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.053+0000 7f8f20a82640 1 -- 172.21.15.59:0/408907229 learned_addr learned my addr 172.21.15.59:0/408907229 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:01.042 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129778691 0 0) 0x7f8f1c145eb0 con 0x7f8f1c0ae500 2024-03-29T11:24:01.042 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.042 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.042 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.042 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:24:01.042 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f0c003520 con 0x7f8f1c0ae500 2024-03-29T11:24:01.042 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441534084 0 0) 0x7f8f1c0a9f00 con 0x7f8f1c0ad0a0 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c145eb0 con 0x7f8f1c0ad0a0 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310303038 0 0) 0x7f8f1c0a94e0 con 0x7f8f1c0add80 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.055+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c0a9f00 con 0x7f8f1c0add80 2024-03-29T11:24:01.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 199747353 0 0) 0x7f8f0c003520 con 0x7f8f1c0ae500 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 >> v1:172.21.15.59:6790/0 conn(0x7f8f1c0add80 legacy=0x7f8f1c0ac990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 >> v1:172.21.15.59:6789/0 conn(0x7f8f1c0ad0a0 legacy=0x7f8f1c151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c0a9670 con 0x7f8f1c0ae500 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8f08001690 con 0x7f8f1c0ae500 2024-03-29T11:24:01.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 10 monclient: dump: 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:01.045 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f080017f0 con 0x7f8f1c0ae500 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 1 -- 172.21.15.59:0/408907229 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8f08001b80 con 0x7f8f1c0ae500 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f1b7fe640 10 monclient: dump: 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.046 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f22ce6640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f22ce6640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.057+0000 7f8f22ce6640 4 set_mon_vals no callback set 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.058+0000 7f8f22ce6640 10 monclient: shutdown 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.058+0000 7f8f22ce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.058+0000 7f8f22ce6640 1 -- 172.21.15.59:0/408907229 >> v1:172.21.15.172:6789/0 conn(0x7f8f1c0ae500 legacy=0x7f8f1c1457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.060+0000 7f8f22ce6640 1 -- 172.21.15.59:0/408907229 shutdown_connections 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.060+0000 7f8f22ce6640 1 -- 172.21.15.59:0/408907229 >> 172.21.15.59:0/408907229 conn(0x7f8f1c0b3c80 msgr2=0x7f8f1c0b4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.060+0000 7f8f22ce6640 1 -- 172.21.15.59:0/408907229 shutdown_connections 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.060+0000 7f8f22ce6640 1 -- 172.21.15.59:0/408907229 wait complete. 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.061+0000 7f8f22ce6640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.061+0000 7f8f22ce6640 10 monclient: monmap: 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:24:01.048 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:24:01.049 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:01.049 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.062+0000 7f8f22ce6640 1 librados: starting msgr at 2024-03-29T11:24:01.049 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.062+0000 7f8f22ce6640 1 librados: starting objecter 2024-03-29T11:24:01.049 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.062+0000 7f8f22ce6640 1 Processor -- start 2024-03-29T11:24:01.049 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.063+0000 7f8f22ce6640 1 -- start start 2024-03-29T11:24:01.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.063+0000 7f8f22ce6640 1 librados: setting wanted keys 2024-03-29T11:24:01.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.063+0000 7f8f22ce6640 1 librados: calling monclient init 2024-03-29T11:24:01.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.063+0000 7f8f22ce6640 10 monclient: init 2024-03-29T11:24:01.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.068+0000 7f8f22ce6640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.068+0000 7f8f22ce6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:01.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.068+0000 7f8f22ce6640 10 monclient(hunting): picked mon.noname-b con 0x7f8f1c0ad0a0 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:01.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.068+0000 7f8f22ce6640 10 monclient(hunting): picked mon.noname-a con 0x7f8f1c0add80 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:01.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.068+0000 7f8f22ce6640 10 monclient(hunting): picked mon.noname-c con 0x7f8f1c0ae500 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:01.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.068+0000 7f8f22ce6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c125ad0 con 0x7f8f1c0add80 2024-03-29T11:24:01.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.068+0000 7f8f22ce6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c124390 con 0x7f8f1c0ad0a0 2024-03-29T11:24:01.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.068+0000 7f8f22ce6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c072380 con 0x7f8f1c0ae500 2024-03-29T11:24:01.051 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.068+0000 7f8f22ce6640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.051 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.068+0000 7f8f22ce6640 10 monclient(hunting): authenticate will time out at 1091.182292s 2024-03-29T11:24:01.053 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.071+0000 7f8f20a82640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8f1c0ad0a0 0x7f8f1c1de940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:33319/0 (socket says 172.21.15.59:33319) 2024-03-29T11:24:01.053 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.071+0000 7f8f20a82640 1 -- 172.21.15.59:0/4161713205 learned_addr learned my addr 172.21.15.59:0/4161713205 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:01.053 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.072+0000 7f8f197fa640 10 client.?.objecter ms_handle_connect 0x7f8f1c0ad0a0 2024-03-29T11:24:01.053 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.072+0000 7f8f197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.072+0000 7f8f197fa640 10 client.?.objecter ms_handle_connect 0x7f8f1c0ae500 2024-03-29T11:24:01.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.072+0000 7f8f197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.072+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487032235 0 0) 0x7f8f1c124390 con 0x7f8f1c0ad0a0 2024-03-29T11:24:01.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.072+0000 7f8f197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.073+0000 7f8f197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.073+0000 7f8f197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.073+0000 7f8f197fa640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:24:01.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.073+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f00004eb0 con 0x7f8f1c0ad0a0 2024-03-29T11:24:01.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.073+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745909000 0 0) 0x7f8f1c072380 con 0x7f8f1c0ae500 2024-03-29T11:24:01.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.073+0000 7f8f197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:24:01.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c124390 con 0x7f8f1c0ae500 2024-03-29T11:24:01.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 10 client.?.objecter ms_handle_connect 0x7f8f1c0add80 2024-03-29T11:24:01.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195754220 0 0) 0x7f8f1c125ad0 con 0x7f8f1c0add80 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c072380 con 0x7f8f1c0add80 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.074+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 228937087 0 0) 0x7f8f1c072380 con 0x7f8f1c0add80 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f1c125ad0 con 0x7f8f1c0add80 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8f14001b50 con 0x7f8f1c0add80 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 10 monclient(hunting): dump: 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:01.057 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3300796777 0 0) 0x7f8f1c124390 con 0x7f8f1c0ae500 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f1c072380 con 0x7f8f1c0ae500 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8f100017d0 con 0x7f8f1c0ae500 2024-03-29T11:24:01.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 10 monclient(hunting): dump: 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 101959069 0 0) 0x7f8f00004eb0 con 0x7f8f1c0ad0a0 2024-03-29T11:24:01.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f1c124390 con 0x7f8f1c0ad0a0 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8f08001b00 con 0x7f8f1c0ad0a0 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 10 monclient(hunting): dump: 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:01.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3109719444 0 0) 0x7f8f1c072380 con 0x7f8f1c0ae500 2024-03-29T11:24:01.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 >> v1:172.21.15.172:6789/0 conn(0x7f8f1c0ad0a0 legacy=0x7f8f1c1de940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 >> v1:172.21.15.59:6789/0 conn(0x7f8f1c0add80 legacy=0x7f8f1c123c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 1 monclient: found mon.c 2024-03-29T11:24:01.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.075+0000 7f8f197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.076+0000 7f8f197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:01.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.076+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c072510 con 0x7f8f1c0ae500 2024-03-29T11:24:01.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.076+0000 7f8f197fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.076+0000 7f8f197fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.076+0000 7f8f197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.064 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.077+0000 7f8f22ce6640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:24:01.064 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.077+0000 7f8f22ce6640 10 monclient: _renew_subs 2024-03-29T11:24:01.064 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.077+0000 7f8f22ce6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:01.064 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.077+0000 7f8f22ce6640 1 -- 172.21.15.59:0/4161713205 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c13f490 con 0x7f8f1c0ae500 2024-03-29T11:24:01.064 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.078+0000 7f8f22ce6640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:01.064 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.078+0000 7f8f22ce6640 10 monclient: _renew_subs 2024-03-29T11:24:01.064 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.078+0000 7f8f22ce6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.078+0000 7f8f22ce6640 1 -- 172.21.15.59:0/4161713205 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c1df4f0 con 0x7f8f1c0ae500 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.078+0000 7f8f22ce6640 1 librados: init done 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.078+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f10001ca0 con 0x7f8f1c0ae500 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.078+0000 7f8f197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.078+0000 7f8f1affd640 4 set_mon_vals no callback set 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.079+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8f10003790 con 0x7f8f1c0ae500 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.079+0000 7f8f197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.079+0000 7f8f197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.079+0000 7f8f197fa640 10 monclient: dump: 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.065 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.079+0000 7f8f22ce6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.079+0000 7f8f22ce6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.079+0000 7f8f22ce6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.079+0000 7f8f22ce6640 1 -- 172.21.15.59:0/4161713205 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8efc005080 con 0x7f8f1c0ae500 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.080+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f8f10019760 con 0x7f8f1c0ae500 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.081+0000 7f8f20a82640 10 monclient: get_auth_request con 0x7f8f0006e0a0 auth_method 0 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.082+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f8f100836c0 con 0x7f8f1c0ae500 2024-03-29T11:24:01.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.082+0000 7f8f197fa640 10 client.4385.objecter ms_dispatch 0x7f8f1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:01.067 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.082+0000 7f8f197fa640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:01.067 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.082+0000 7f8f197fa640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:01.067 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.082+0000 7f8f197fa640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T11:24:01.067 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.082+0000 7f8f197fa640 10 client.4385.objecter ms_handle_connect 0x7f8f0006e0a0 2024-03-29T11:24:01.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.092+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f1004efa0 con 0x7f8f1c0ae500 2024-03-29T11:24:01.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.092+0000 7f8f197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.092+0000 7f8f197fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:01.403 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.423+0000 7f8f22ce6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.403 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.423+0000 7f8f22ce6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.403 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.423+0000 7f8f22ce6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:01.403 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.423+0000 7f8f22ce6640 1 -- 172.21.15.59:0/4161713205 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8efc005330 con 0x7f8f1c0ae500 2024-03-29T11:24:01.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.426+0000 7f8f197fa640 1 -- 172.21.15.59:0/4161713205 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f8f10050ed0 con 0x7f8f1c0ae500 2024-03-29T11:24:01.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.426+0000 7f8f197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.426+0000 7f8f197fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:01.406 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:24:01.407 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:01.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.429+0000 7f8f22ce6640 10 librados: watch_flush enter 2024-03-29T11:24:01.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.431+0000 7f8f22ce6640 10 librados: watch_flush exit 2024-03-29T11:24:01.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.431+0000 7f8f22ce6640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T11:24:01.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.431+0000 7f8f22ce6640 10 client.4385.objecter successfully canceled tick 2024-03-29T11:24:01.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.431+0000 7f8f22ce6640 1 -- 172.21.15.59:0/4161713205 >> v1:172.21.15.172:6816/2420742773 conn(0x7f8f0006e0a0 legacy=0x7f8f00070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.431+0000 7f8f22ce6640 10 monclient: shutdown 2024-03-29T11:24:01.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.431+0000 7f8f22ce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.431+0000 7f8f22ce6640 1 -- 172.21.15.59:0/4161713205 >> v1:172.21.15.59:6790/0 conn(0x7f8f1c0ae500 legacy=0x7f8f1c1e5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.432+0000 7f8f22ce6640 1 -- 172.21.15.59:0/4161713205 shutdown_connections 2024-03-29T11:24:01.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.432+0000 7f8f22ce6640 1 -- 172.21.15.59:0/4161713205 >> 172.21.15.59:0/4161713205 conn(0x7f8f1c0b3c80 msgr2=0x7f8f1c0ac930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.432+0000 7f8f22ce6640 1 -- 172.21.15.59:0/4161713205 shutdown_connections 2024-03-29T11:24:01.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.432+0000 7f8f22ce6640 1 -- 172.21.15.59:0/4161713205 wait complete. 2024-03-29T11:24:01.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:01.432+0000 7f8f22ce6640 1 librados: shutdown 2024-03-29T11:24:01.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:08.236 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:08.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.412+0000 7f137c664640 10 monclient: get_monmap_and_config 2024-03-29T11:24:08.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.412+0000 7f137c664640 10 monclient: build_initial_monmap 2024-03-29T11:24:08.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.413+0000 7f137c664640 10 monclient: monmap: 2024-03-29T11:24:08.394 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:08.394 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:08.394 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:24:08.413725+0000 2024-03-29T11:24:08.394 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:24:08.413725+0000 2024-03-29T11:24:08.394 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:08.394 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:08.394 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:08.395 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:24:08.395 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:24:08.395 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:08.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.414+0000 7f137c664640 1 Processor -- start 2024-03-29T11:24:08.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.414+0000 7f137c664640 1 -- start start 2024-03-29T11:24:08.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.414+0000 7f137c664640 10 monclient: init 2024-03-29T11:24:08.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.415+0000 7f137c664640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:08.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.415+0000 7f137c664640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:08.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.415+0000 7f137c664640 10 monclient(hunting): picked mon.noname-b con 0x7f13740b0770 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:08.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.415+0000 7f137c664640 10 monclient(hunting): picked mon.noname-a con 0x7f137414a700 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:08.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.415+0000 7f137c664640 10 monclient(hunting): picked mon.noname-c con 0x7f13740ac5a0 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:08.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.415+0000 7f137c664640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13740aba70 con 0x7f137414a700 2024-03-29T11:24:08.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.415+0000 7f137c664640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13740aecb0 con 0x7f13740ac5a0 2024-03-29T11:24:08.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.415+0000 7f137c664640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13740af150 con 0x7f13740b0770 2024-03-29T11:24:08.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.415+0000 7f137c664640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:08.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.415+0000 7f137c664640 10 monclient(hunting): authenticate will time out at 1098.529499s 2024-03-29T11:24:08.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.416+0000 7f137ac01640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f13740ac5a0 0x7f1374151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:43070/0 (socket says 172.21.15.59:43070) 2024-03-29T11:24:08.398 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.416+0000 7f137ac01640 1 -- 172.21.15.59:0/2425070680 learned_addr learned my addr 172.21.15.59:0/2425070680 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:08.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.418+0000 7f13793fe640 1 -- 172.21.15.59:0/2425070680 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135078543 0 0) 0x7f13740aecb0 con 0x7f13740ac5a0 2024-03-29T11:24:08.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.418+0000 7f13793fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.418+0000 7f13793fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.418+0000 7f13793fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.418+0000 7f13793fe640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:24:08.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.418+0000 7f13793fe640 1 -- 172.21.15.59:0/2425070680 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1360003520 con 0x7f13740ac5a0 2024-03-29T11:24:08.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 1 -- 172.21.15.59:0/2425070680 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 9338700 0 0) 0x7f1360003520 con 0x7f13740ac5a0 2024-03-29T11:24:08.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 1 -- 172.21.15.59:0/2425070680 >> v1:172.21.15.59:6790/0 conn(0x7f13740b0770 legacy=0x7f1374149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.400 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 1 -- 172.21.15.59:0/2425070680 >> v1:172.21.15.59:6789/0 conn(0x7f137414a700 legacy=0x7f13740abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 1 monclient: found mon.noname-c 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 1 -- 172.21.15.59:0/2425070680 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13740ac9d0 con 0x7f13740ac5a0 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 1 -- 172.21.15.59:0/2425070680 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f136c001a30 con 0x7f13740ac5a0 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 10 monclient: dump: 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:08.401 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:08.402 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:08.402 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.402 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:08.402 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:08.402 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:08.402 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:08.402 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:08.402 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f137c664640 5 monclient: authenticate success, global_id 4372 2024-03-29T11:24:08.402 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f137c664640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:08.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 1 -- 172.21.15.59:0/2425070680 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f136c001cc0 con 0x7f13740ac5a0 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 1 -- 172.21.15.59:0/2425070680 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f136c002090 con 0x7f13740ac5a0 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f13793fe640 10 monclient: dump: 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:08.406 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f137c664640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.419+0000 7f137c664640 4 set_mon_vals no callback set 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.421+0000 7f137c664640 10 monclient: shutdown 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.421+0000 7f137c664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.421+0000 7f137c664640 1 -- 172.21.15.59:0/2425070680 >> v1:172.21.15.172:6789/0 conn(0x7f13740ac5a0 legacy=0x7f1374151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.421+0000 7f137c664640 1 -- 172.21.15.59:0/2425070680 shutdown_connections 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.421+0000 7f137c664640 1 -- 172.21.15.59:0/2425070680 >> 172.21.15.59:0/2425070680 conn(0x7f137406b660 msgr2=0x7f137406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.421+0000 7f137c664640 1 -- 172.21.15.59:0/2425070680 shutdown_connections 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.421+0000 7f137c664640 1 -- 172.21.15.59:0/2425070680 wait complete. 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.422+0000 7f137c664640 10 monclient: build_initial_monmap 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.422+0000 7f137c664640 10 monclient: monmap: 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:08.407 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.422+0000 7f137c664640 1 librados: starting msgr at 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.422+0000 7f137c664640 1 librados: starting objecter 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.422+0000 7f137c664640 1 Processor -- start 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.423+0000 7f137c664640 1 -- start start 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.423+0000 7f137c664640 1 librados: setting wanted keys 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.423+0000 7f137c664640 1 librados: calling monclient init 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.423+0000 7f137c664640 10 monclient: init 2024-03-29T11:24:08.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.424+0000 7f137c664640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:08.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.424+0000 7f137c664640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:08.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.424+0000 7f137c664640 10 monclient(hunting): picked mon.noname-a con 0x7f13740ac5a0 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:08.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.424+0000 7f137c664640 10 monclient(hunting): picked mon.noname-b con 0x7f13740b0770 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:08.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.424+0000 7f137c664640 10 monclient(hunting): picked mon.noname-c con 0x7f137414a700 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:08.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.424+0000 7f137c664640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1374122830 con 0x7f13740ac5a0 2024-03-29T11:24:08.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.424+0000 7f137c664640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13741e7dd0 con 0x7f13740b0770 2024-03-29T11:24:08.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.424+0000 7f137c664640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1374122a60 con 0x7f137414a700 2024-03-29T11:24:08.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.424+0000 7f137c664640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:08.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.424+0000 7f137c664640 10 monclient(hunting): authenticate will time out at 1098.537879s 2024-03-29T11:24:08.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.427+0000 7f1379bff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f13740b0770 0x7f1374120400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:44718/0 (socket says 172.21.15.59:44718) 2024-03-29T11:24:08.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.427+0000 7f1379bff640 1 -- 172.21.15.59:0/4141752530 learned_addr learned my addr 172.21.15.59:0/4141752530 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:08.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.427+0000 7f1366ffd640 10 client.?.objecter ms_handle_connect 0x7f13740b0770 2024-03-29T11:24:08.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.427+0000 7f1366ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:08.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.428+0000 7f1366ffd640 10 client.?.objecter ms_handle_connect 0x7f137414a700 2024-03-29T11:24:08.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.428+0000 7f1366ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:08.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.429+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404429470 0 0) 0x7f13741e7dd0 con 0x7f13740b0770 2024-03-29T11:24:08.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.429+0000 7f1366ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.429+0000 7f1366ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.429+0000 7f1366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.429+0000 7f1366ffd640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:24:08.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1358004eb0 con 0x7f13740b0770 2024-03-29T11:24:08.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949107696 0 0) 0x7f1374122a60 con 0x7f137414a700 2024-03-29T11:24:08.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13741e7dd0 con 0x7f137414a700 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1032488401 0 0) 0x7f1358004eb0 con 0x7f13740b0770 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1374122a60 con 0x7f13740b0770 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f13680017d0 con 0x7f13740b0770 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.415 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 10 client.?.objecter ms_handle_connect 0x7f13740ac5a0 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.430+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2689786483 0 0) 0x7f13741e7dd0 con 0x7f137414a700 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1358004eb0 con 0x7f137414a700 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f136c001850 con 0x7f137414a700 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:08.416 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987619589 0 0) 0x7f1374122830 con 0x7f13740ac5a0 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13741e7dd0 con 0x7f13740ac5a0 2024-03-29T11:24:08.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 376752199 0 0) 0x7f1358004eb0 con 0x7f137414a700 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 >> v1:172.21.15.172:6789/0 conn(0x7f13740b0770 legacy=0x7f1374120400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 >> v1:172.21.15.59:6789/0 conn(0x7f13740ac5a0 legacy=0x7f13741e7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 1 monclient: found mon.c 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13741e8260 con 0x7f137414a700 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f1366ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f137c664640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f137c664640 10 monclient: _renew_subs 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f137c664640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:08.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.431+0000 7f137c664640 1 -- 172.21.15.59:0/4141752530 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13741231f0 con 0x7f137414a700 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f137c664640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f137c664640 10 monclient: _renew_subs 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f137c664640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f137c664640 1 -- 172.21.15.59:0/4141752530 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13741eff20 con 0x7f137414a700 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f137c664640 1 librados: init done 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f136c0015e0 con 0x7f137414a700 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f1366ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f136c004ee0 con 0x7f137414a700 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f1366ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f1366ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f1366ffd640 10 monclient: dump: 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:08.419 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:08.420 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.420 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:08.420 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:08.420 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:08.420 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:08.420 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:08.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.432+0000 7f1378bfd640 4 set_mon_vals no callback set 2024-03-29T11:24:08.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.433+0000 7f137c664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.434+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f136c005040 con 0x7f137414a700 2024-03-29T11:24:08.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.435+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f136c084d70 con 0x7f137414a700 2024-03-29T11:24:08.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.435+0000 7f1366ffd640 10 client.4388.objecter ms_dispatch 0x7f1374000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:08.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.435+0000 7f1366ffd640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:08.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.435+0000 7f1366ffd640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:08.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.435+0000 7f137c664640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.435+0000 7f137c664640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:08.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.435+0000 7f137c664640 1 -- 172.21.15.59:0/4141752530 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c005080 con 0x7f137414a700 2024-03-29T11:24:08.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.436+0000 7f137a400640 10 monclient: get_auth_request con 0x7f135806e000 auth_method 0 2024-03-29T11:24:08.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.436+0000 7f1366ffd640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-29T11:24:08.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.437+0000 7f1366ffd640 10 client.4388.objecter ms_handle_connect 0x7f135806e000 2024-03-29T11:24:08.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.446+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f136c050480 con 0x7f137414a700 2024-03-29T11:24:08.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.446+0000 7f1366ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.446+0000 7f1366ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:08.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.781+0000 7f137c664640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:08.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.781+0000 7f137c664640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:08.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.781+0000 7f137c664640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:08.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.781+0000 7f137c664640 1 -- 172.21.15.59:0/4141752530 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f135c005330 con 0x7f137414a700 2024-03-29T11:24:08.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.781+0000 7f1366ffd640 1 -- 172.21.15.59:0/4141752530 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f136c052580 con 0x7f137414a700 2024-03-29T11:24:08.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.782+0000 7f1366ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:08.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.782+0000 7f1366ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:08.763 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:24:08.763 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:08.766 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.785+0000 7f137c664640 10 librados: watch_flush enter 2024-03-29T11:24:08.766 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.785+0000 7f137c664640 10 librados: watch_flush exit 2024-03-29T11:24:08.766 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.785+0000 7f137c664640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-29T11:24:08.766 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.785+0000 7f137c664640 10 client.4388.objecter successfully canceled tick 2024-03-29T11:24:08.766 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.785+0000 7f137c664640 1 -- 172.21.15.59:0/4141752530 >> v1:172.21.15.172:6816/2420742773 conn(0x7f135806e000 legacy=0x7f13580704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.766 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.785+0000 7f137c664640 10 monclient: shutdown 2024-03-29T11:24:08.766 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.785+0000 7f137c664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:08.767 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.785+0000 7f137c664640 1 -- 172.21.15.59:0/4141752530 >> v1:172.21.15.59:6790/0 conn(0x7f137414a700 legacy=0x7f13741ee800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.767 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.786+0000 7f137c664640 1 -- 172.21.15.59:0/4141752530 shutdown_connections 2024-03-29T11:24:08.767 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.786+0000 7f137c664640 1 -- 172.21.15.59:0/4141752530 >> 172.21.15.59:0/4141752530 conn(0x7f137406b660 msgr2=0x7f13740b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:08.767 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.787+0000 7f137c664640 1 -- 172.21.15.59:0/4141752530 shutdown_connections 2024-03-29T11:24:08.767 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.787+0000 7f137c664640 1 -- 172.21.15.59:0/4141752530 wait complete. 2024-03-29T11:24:08.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:08.788+0000 7f137c664640 1 librados: shutdown 2024-03-29T11:24:08.785 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:15.591 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:15.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.755+0000 7ffaff10e640 10 monclient: get_monmap_and_config 2024-03-29T11:24:15.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.755+0000 7ffaff10e640 10 monclient: build_initial_monmap 2024-03-29T11:24:15.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.755+0000 7ffaff10e640 10 monclient: monmap: 2024-03-29T11:24:15.737 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:15.737 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:15.737 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:24:15.756446+0000 2024-03-29T11:24:15.737 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:24:15.756446+0000 2024-03-29T11:24:15.737 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:15.737 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:15.737 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:15.737 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:24:15.737 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:24:15.737 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:15.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.756+0000 7ffaff10e640 1 Processor -- start 2024-03-29T11:24:15.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.756+0000 7ffaff10e640 1 -- start start 2024-03-29T11:24:15.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.756+0000 7ffaff10e640 10 monclient: init 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.757+0000 7ffaff10e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.757+0000 7ffaff10e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.757+0000 7ffaff10e640 10 monclient(hunting): picked mon.noname-c con 0x7ffaf81422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.757+0000 7ffaff10e640 10 monclient(hunting): picked mon.noname-b con 0x7ffaf8142690 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaff10e640 10 monclient(hunting): picked mon.noname-a con 0x7ffaf80b17d0 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaff10e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf8142c80 con 0x7ffaf80b17d0 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffafceaa640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ffaf81422b0 0x7ffaf813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:52826/0 (socket says 172.21.15.59:52826) 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffafceaa640 1 -- 172.21.15.59:0/897427929 learned_addr learned my addr 172.21.15.59:0/897427929 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaff10e640 1 -- 172.21.15.59:0/897427929 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf80aae40 con 0x7ffaf81422b0 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaff10e640 1 -- 172.21.15.59:0/897427929 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf813fbd0 con 0x7ffaf8142690 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaff10e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaff10e640 10 monclient(hunting): authenticate will time out at 1105.872041s 2024-03-29T11:24:15.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383968830 0 0) 0x7ffaf80aae40 con 0x7ffaf81422b0 2024-03-29T11:24:15.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae8003520 con 0x7ffaf81422b0 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112652905 0 0) 0x7ffaf813fbd0 con 0x7ffaf8142690 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.758+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf80aae40 con 0x7ffaf8142690 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574270339 0 0) 0x7ffaf8142c80 con 0x7ffaf80b17d0 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:24:15.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf813fbd0 con 0x7ffaf80b17d0 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4180021159 0 0) 0x7ffae8003520 con 0x7ffaf81422b0 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 >> v1:172.21.15.59:6790/0 conn(0x7ffaf8142690 legacy=0x7ffaf80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 >> v1:172.21.15.59:6789/0 conn(0x7ffaf80b17d0 legacy=0x7ffaf80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 1 monclient: found mon.noname-c 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf814f920 con 0x7ffaf81422b0 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ffaec001690 con 0x7ffaf81422b0 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:15.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient: dump: 2024-03-29T11:24:15.748 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:15.748 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:15.748 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:15.748 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:15.748 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.748 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:15.748 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:15.748 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:15.748 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaff10e640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaff10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffaec0017f0 con 0x7ffaf81422b0 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 1 -- 172.21.15.59:0/897427929 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ffaec001bc0 con 0x7ffaf81422b0 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.759+0000 7ffaf77fe640 10 monclient: dump: 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.749 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.760+0000 7ffaff10e640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.760+0000 7ffaff10e640 4 set_mon_vals no callback set 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.760+0000 7ffaff10e640 10 monclient: shutdown 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.761+0000 7ffaff10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.761+0000 7ffaff10e640 1 -- 172.21.15.59:0/897427929 >> v1:172.21.15.172:6789/0 conn(0x7ffaf81422b0 legacy=0x7ffaf813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.761+0000 7ffaff10e640 1 -- 172.21.15.59:0/897427929 shutdown_connections 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.761+0000 7ffaff10e640 1 -- 172.21.15.59:0/897427929 >> 172.21.15.59:0/897427929 conn(0x7ffaf8066a00 msgr2=0x7ffaf8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.761+0000 7ffaff10e640 1 -- 172.21.15.59:0/897427929 shutdown_connections 2024-03-29T11:24:15.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.761+0000 7ffaff10e640 1 -- 172.21.15.59:0/897427929 wait complete. 2024-03-29T11:24:15.756 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.763+0000 7ffaff10e640 10 monclient: build_initial_monmap 2024-03-29T11:24:15.756 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.763+0000 7ffaff10e640 10 monclient: monmap: 2024-03-29T11:24:15.756 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:15.756 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.763+0000 7ffaff10e640 1 librados: starting msgr at 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.763+0000 7ffaff10e640 1 librados: starting objecter 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.763+0000 7ffaff10e640 1 Processor -- start 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.763+0000 7ffaff10e640 1 -- start start 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.763+0000 7ffaff10e640 1 librados: setting wanted keys 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.763+0000 7ffaff10e640 1 librados: calling monclient init 2024-03-29T11:24:15.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.763+0000 7ffaff10e640 10 monclient: init 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.765+0000 7ffaff10e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.765+0000 7ffaff10e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.765+0000 7ffaff10e640 10 monclient(hunting): picked mon.noname-b con 0x7ffaf80b17d0 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.765+0000 7ffaff10e640 10 monclient(hunting): picked mon.noname-c con 0x7ffaf81422b0 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaf7fff640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7ffaf81422b0 0x7ffaf80b0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:52288/0 (socket says 172.21.15.59:52288) 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaf7fff640 1 -- 172.21.15.59:0/1992125585 learned_addr learned my addr 172.21.15.59:0/1992125585 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaff10e640 10 monclient(hunting): picked mon.noname-a con 0x7ffaf8142690 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaf57fa640 10 client.?.objecter ms_handle_connect 0x7ffaf81422b0 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaf57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf80aeea0 con 0x7ffaf8142690 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf80af180 con 0x7ffaf80b17d0 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf80ad730 con 0x7ffaf81422b0 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaff10e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:15.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaff10e640 10 monclient(hunting): authenticate will time out at 1105.880407s 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaf57fa640 10 client.?.objecter ms_handle_connect 0x7ffaf8142690 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.766+0000 7ffaf57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495023982 0 0) 0x7ffaf80ad730 con 0x7ffaf81422b0 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffadc004eb0 con 0x7ffaf81422b0 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153509205 0 0) 0x7ffaf80aeea0 con 0x7ffaf8142690 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf80ad730 con 0x7ffaf8142690 2024-03-29T11:24:15.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 839480516 0 0) 0x7ffaf80ad730 con 0x7ffaf8142690 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaf80aeea0 con 0x7ffaf8142690 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ffaf0001bf0 con 0x7ffaf8142690 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): dump: 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:15.760 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2616639966 0 0) 0x7ffadc004eb0 con 0x7ffaf81422b0 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaf80ad730 con 0x7ffaf81422b0 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ffae4001900 con 0x7ffaf81422b0 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.767+0000 7ffaf57fa640 10 monclient(hunting): dump: 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:15.761 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.768+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3687141685 0 0) 0x7ffaf80aeea0 con 0x7ffaf8142690 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.768+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 >> v1:172.21.15.59:6790/0 conn(0x7ffaf81422b0 legacy=0x7ffaf80b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 >> v1:172.21.15.172:6789/0 conn(0x7ffaf80b17d0 legacy=0x7ffaf80ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaf57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaf57fa640 1 monclient: found mon.a 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaf57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaf57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf80ad3c0 con 0x7ffaf8142690 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaf57fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:15.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaf57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaf57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaf57fa640 10 client.?.objecter ms_handle_connect 0x7ffaf80b17d0 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaf57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaff10e640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaff10e640 10 monclient: _renew_subs 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaff10e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf80b13b0 con 0x7ffaf8142690 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaff10e640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaff10e640 10 monclient: _renew_subs 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaff10e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf80ad170 con 0x7ffaf8142690 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.769+0000 7ffaff10e640 1 librados: init done 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.770+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffaf0003970 con 0x7ffaf8142690 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.770+0000 7ffaf57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:15.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.770+0000 7ffaf6ffd640 4 set_mon_vals no callback set 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.771+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ffaf0003c80 con 0x7ffaf8142690 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.771+0000 7ffaf57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.771+0000 7ffaf57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.771+0000 7ffaf57fa640 10 monclient: dump: 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:15.764 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.772+0000 7ffaff10e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:15.765 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.772+0000 7ffaff10e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:15.765 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.772+0000 7ffaff10e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:15.765 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.772+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad8005080 con 0x7ffaf8142690 2024-03-29T11:24:15.765 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.772+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7ffaf0019c50 con 0x7ffaf8142690 2024-03-29T11:24:15.765 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.773+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7ffaf0083ae0 con 0x7ffaf8142690 2024-03-29T11:24:15.765 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.773+0000 7ffaf57fa640 10 client.4401.objecter ms_dispatch 0x7ffaf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:15.765 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.773+0000 7ffafceaa640 10 monclient: get_auth_request con 0x7ffadc06dea0 auth_method 0 2024-03-29T11:24:15.765 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.773+0000 7ffaf57fa640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:15.765 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.773+0000 7ffaf57fa640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:15.765 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.773+0000 7ffaf57fa640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-29T11:24:15.765 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.773+0000 7ffaf57fa640 10 client.4401.objecter ms_handle_connect 0x7ffadc06dea0 2024-03-29T11:24:15.766 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.784+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffaf004f3c0 con 0x7ffaf8142690 2024-03-29T11:24:15.766 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.785+0000 7ffaf57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:15.766 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:15.785+0000 7ffaf57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:16.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.112+0000 7ffaff10e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.112+0000 7ffaff10e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.112+0000 7ffaff10e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:16.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.112+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffad8005330 con 0x7ffaf8142690 2024-03-29T11:24:16.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.113+0000 7ffaf57fa640 1 -- 172.21.15.59:0/1992125585 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7ffaf00512f0 con 0x7ffaf8142690 2024-03-29T11:24:16.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.113+0000 7ffaf57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.113+0000 7ffaf57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:16.095 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:24:16.095 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:16.098 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.116+0000 7ffaff10e640 10 librados: watch_flush enter 2024-03-29T11:24:16.098 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.116+0000 7ffaff10e640 10 librados: watch_flush exit 2024-03-29T11:24:16.098 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.116+0000 7ffaff10e640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-29T11:24:16.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.116+0000 7ffaff10e640 10 client.4401.objecter successfully canceled tick 2024-03-29T11:24:16.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.117+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 >> v1:172.21.15.172:6816/2420742773 conn(0x7ffadc06dea0 legacy=0x7ffadc070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.117+0000 7ffaff10e640 10 monclient: shutdown 2024-03-29T11:24:16.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.117+0000 7ffaff10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:16.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.117+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 >> v1:172.21.15.59:6789/0 conn(0x7ffaf8142690 legacy=0x7ffaf81e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.117+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 shutdown_connections 2024-03-29T11:24:16.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.117+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 >> 172.21.15.59:0/1992125585 conn(0x7ffaf8066a00 msgr2=0x7ffaf813d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:16.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.117+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 shutdown_connections 2024-03-29T11:24:16.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.117+0000 7ffaff10e640 1 -- 172.21.15.59:0/1992125585 wait complete. 2024-03-29T11:24:16.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:16.117+0000 7ffaff10e640 1 librados: shutdown 2024-03-29T11:24:16.114 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:22.921 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:23.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.071+0000 7f4799063640 10 monclient: get_monmap_and_config 2024-03-29T11:24:23.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.071+0000 7f4799063640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.071+0000 7f4799063640 10 monclient: monmap: 2024-03-29T11:24:23.054 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:23.054 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:23.054 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:24:23.073495+0000 2024-03-29T11:24:23.054 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:24:23.073495+0000 2024-03-29T11:24:23.054 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.054 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:23.054 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:23.054 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:24:23.055 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:24:23.055 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:23.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.072+0000 7f4799063640 1 Processor -- start 2024-03-29T11:24:23.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.072+0000 7f4799063640 1 -- start start 2024-03-29T11:24:23.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.072+0000 7f4799063640 10 monclient: init 2024-03-29T11:24:23.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.073+0000 7f4799063640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.073+0000 7f4799063640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:23.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.073+0000 7f4799063640 10 monclient(hunting): picked mon.noname-a con 0x7f47941457f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:23.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.073+0000 7f4799063640 10 monclient(hunting): picked mon.noname-b con 0x7f479413f650 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:23.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.073+0000 7f4799063640 10 monclient(hunting): picked mon.noname-c con 0x7f47940a9d70 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:23.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.073+0000 7f4799063640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47940aaef0 con 0x7f47941457f0 2024-03-29T11:24:23.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.073+0000 7f4799063640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4794145bd0 con 0x7f47940a9d70 2024-03-29T11:24:23.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.073+0000 7f4799063640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4794145eb0 con 0x7f479413f650 2024-03-29T11:24:23.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.073+0000 7f4799063640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.073+0000 7f4799063640 10 monclient(hunting): authenticate will time out at 1113.188322s 2024-03-29T11:24:23.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.074+0000 7f479359e640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f47940a9d70 0x7f479414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:38484/0 (socket says 172.21.15.59:38484) 2024-03-29T11:24:23.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.074+0000 7f479359e640 1 -- 172.21.15.59:0/3906338902 learned_addr learned my addr 172.21.15.59:0/3906338902 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:23.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.074+0000 7f479259c640 1 -- 172.21.15.59:0/3906338902 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459442723 0 0) 0x7f4794145eb0 con 0x7f479413f650 2024-03-29T11:24:23.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.074+0000 7f479259c640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.074+0000 7f479259c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.074+0000 7f479259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.074+0000 7f479259c640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:24:23.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.074+0000 7f479259c640 1 -- 172.21.15.59:0/3906338902 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4780003520 con 0x7f479413f650 2024-03-29T11:24:23.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.074+0000 7f479259c640 1 -- 172.21.15.59:0/3906338902 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 817835611 0 0) 0x7f4780003520 con 0x7f479413f650 2024-03-29T11:24:23.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.074+0000 7f479259c640 1 -- 172.21.15.59:0/3906338902 >> v1:172.21.15.172:6789/0 conn(0x7f47940a9d70 legacy=0x7f479414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.075+0000 7f479259c640 1 -- 172.21.15.59:0/3906338902 >> v1:172.21.15.59:6789/0 conn(0x7f47941457f0 legacy=0x7f479413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:23.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 1 monclient: found mon.noname-b 2024-03-29T11:24:23.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:24:23.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 1 -- 172.21.15.59:0/3906338902 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4794146340 con 0x7f479413f650 2024-03-29T11:24:23.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 1 -- 172.21.15.59:0/3906338902 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f477c001690 con 0x7f479413f650 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f479259c640 10 monclient: dump: 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f4799063640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:24:23.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.076+0000 7f4799063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.077+0000 7f479259c640 1 -- 172.21.15.59:0/3906338902 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f477c001920 con 0x7f479413f650 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.077+0000 7f479259c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.077+0000 7f479259c640 1 -- 172.21.15.59:0/3906338902 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f477c001d30 con 0x7f479413f650 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.077+0000 7f479259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.077+0000 7f479259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.077+0000 7f479259c640 10 monclient: dump: 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:23.062 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.077+0000 7f4799063640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.077+0000 7f4799063640 4 set_mon_vals no callback set 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.077+0000 7f4799063640 10 monclient: shutdown 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.077+0000 7f4799063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.077+0000 7f4799063640 1 -- 172.21.15.59:0/3906338902 >> v1:172.21.15.59:6790/0 conn(0x7f479413f650 legacy=0x7f47940b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.078+0000 7f4799063640 1 -- 172.21.15.59:0/3906338902 shutdown_connections 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.078+0000 7f4799063640 1 -- 172.21.15.59:0/3906338902 >> 172.21.15.59:0/3906338902 conn(0x7f4794066a00 msgr2=0x7f4794066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.078+0000 7f4799063640 1 -- 172.21.15.59:0/3906338902 shutdown_connections 2024-03-29T11:24:23.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.078+0000 7f4799063640 1 -- 172.21.15.59:0/3906338902 wait complete. 2024-03-29T11:24:23.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.079+0000 7f4799063640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.079+0000 7f4799063640 10 monclient: monmap: 2024-03-29T11:24:23.069 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:23.069 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:23.069 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:24:23.069 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:24:23.069 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.069 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:23.069 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:23.069 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:24:23.069 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.079+0000 7f4799063640 1 librados: starting msgr at 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.079+0000 7f4799063640 1 librados: starting objecter 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.079+0000 7f4799063640 1 Processor -- start 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.079+0000 7f4799063640 1 -- start start 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.079+0000 7f4799063640 1 librados: setting wanted keys 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 1 librados: calling monclient init 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 10 monclient: init 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 10 monclient(hunting): picked mon.noname-c con 0x7f47940a9d70 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 10 monclient(hunting): picked mon.noname-b con 0x7f479413f650 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 10 monclient(hunting): picked mon.noname-a con 0x7f47941457f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:23.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47941df3c0 con 0x7f47941457f0 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47941dbf40 con 0x7f479413f650 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47940b0f90 con 0x7f47940a9d70 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.080+0000 7f4799063640 10 monclient(hunting): authenticate will time out at 1113.195466s 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.081+0000 7f478bfff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f479413f650 0x7f47940b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:39428/0 (socket says 172.21.15.59:39428) 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.081+0000 7f478bfff640 1 -- 172.21.15.59:0/771945363 learned_addr learned my addr 172.21.15.59:0/771945363 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.081+0000 7f478b7fe640 10 client.?.objecter ms_handle_connect 0x7f47940a9d70 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.081+0000 7f478b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169316127 0 0) 0x7f47940b0f90 con 0x7f47940a9d70 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4774004eb0 con 0x7f47940a9d70 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 10 client.?.objecter ms_handle_connect 0x7f479413f650 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1086333871 0 0) 0x7f4774004eb0 con 0x7f47940a9d70 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4774005300 con 0x7f47940a9d70 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4784001790 con 0x7f47940a9d70 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:23.072 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.082+0000 7f478b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.083+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3246059046 0 0) 0x7f4774005300 con 0x7f47940a9d70 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.083+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 >> v1:172.21.15.172:6789/0 conn(0x7f479413f650 legacy=0x7f47940b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.083+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 >> v1:172.21.15.59:6789/0 conn(0x7f47941457f0 legacy=0x7f47941e7070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 1 monclient: found mon.c 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47940b1120 con 0x7f47940a9d70 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f4799063640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f4799063640 10 monclient: _renew_subs 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f4799063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f4799063640 1 -- 172.21.15.59:0/771945363 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47941df8e0 con 0x7f47940a9d70 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f4799063640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f4799063640 10 monclient: _renew_subs 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f4799063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f4799063640 1 -- 172.21.15.59:0/771945363 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4794142ba0 con 0x7f47940a9d70 2024-03-29T11:24:23.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f4799063640 1 librados: init done 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4784001d20 con 0x7f47940a9d70 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4784001790 con 0x7f47940a9d70 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f478b7fe640 10 monclient: dump: 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:23.075 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.084+0000 7f4791d9b640 4 set_mon_vals no callback set 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.086+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f47840194e0 con 0x7f47940a9d70 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.086+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f4784083520 con 0x7f47940a9d70 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.087+0000 7f478b7fe640 10 client.4400.objecter ms_dispatch 0x7f4794000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.087+0000 7f478b7fe640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.087+0000 7f478b7fe640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.087+0000 7f4799063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.087+0000 7f4799063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.087+0000 7f4799063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.087+0000 7f478b7fe640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.087+0000 7f4799063640 1 -- 172.21.15.59:0/771945363 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4778005080 con 0x7f47940a9d70 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.087+0000 7f478bfff640 10 monclient: get_auth_request con 0x7f477406e0e0 auth_method 0 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.087+0000 7f478b7fe640 10 client.4400.objecter ms_handle_connect 0x7f477406e0e0 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.095+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f478404ec30 con 0x7f47940a9d70 2024-03-29T11:24:23.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.095+0000 7f478b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.078 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.095+0000 7f478b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:23.391 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.409+0000 7f4799063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.391 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.409+0000 7f4799063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.409+0000 7f4799063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:23.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.409+0000 7f4799063640 1 -- 172.21.15.59:0/771945363 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4778005330 con 0x7f47940a9d70 2024-03-29T11:24:23.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.410+0000 7f478b7fe640 1 -- 172.21.15.59:0/771945363 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f4784050d30 con 0x7f47940a9d70 2024-03-29T11:24:23.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.410+0000 7f478b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.410+0000 7f478b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:23.392 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:24:23.393 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:23.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.413+0000 7f4799063640 10 librados: watch_flush enter 2024-03-29T11:24:23.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.413+0000 7f4799063640 10 librados: watch_flush exit 2024-03-29T11:24:23.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.413+0000 7f4799063640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-29T11:24:23.396 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.413+0000 7f4799063640 10 client.4400.objecter successfully canceled tick 2024-03-29T11:24:23.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.414+0000 7f4799063640 1 -- 172.21.15.59:0/771945363 >> v1:172.21.15.172:6816/2420742773 conn(0x7f477406e0e0 legacy=0x7f47740705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.415+0000 7f4799063640 10 monclient: shutdown 2024-03-29T11:24:23.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.415+0000 7f4799063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.415+0000 7f4799063640 1 -- 172.21.15.59:0/771945363 >> v1:172.21.15.59:6790/0 conn(0x7f47940a9d70 legacy=0x7f47941db830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.415+0000 7f4799063640 1 -- 172.21.15.59:0/771945363 shutdown_connections 2024-03-29T11:24:23.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.415+0000 7f4799063640 1 -- 172.21.15.59:0/771945363 >> 172.21.15.59:0/771945363 conn(0x7f4794066a00 msgr2=0x7f479413dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.398 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.416+0000 7f4799063640 1 -- 172.21.15.59:0/771945363 shutdown_connections 2024-03-29T11:24:23.398 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.416+0000 7f4799063640 1 -- 172.21.15.59:0/771945363 wait complete. 2024-03-29T11:24:23.398 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:23.416+0000 7f4799063640 1 librados: shutdown 2024-03-29T11:24:23.412 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:30.220 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:30.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.388+0000 7f4494055640 10 monclient: get_monmap_and_config 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.388+0000 7f4494055640 10 monclient: build_initial_monmap 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.388+0000 7f4494055640 10 monclient: monmap: 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:24:30.390310+0000 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:24:30.390310+0000 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.390+0000 7f4494055640 1 Processor -- start 2024-03-29T11:24:30.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.390+0000 7f4494055640 1 -- start start 2024-03-29T11:24:30.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.390+0000 7f4494055640 10 monclient: init 2024-03-29T11:24:30.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4494055640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4494055640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4494055640 10 monclient(hunting): picked mon.noname-c con 0x7f448c0acf60 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4494055640 10 monclient(hunting): picked mon.noname-a con 0x7f448c13ff40 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4494055640 10 monclient(hunting): picked mon.noname-b con 0x7f448c1403f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4494055640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c0b4310 con 0x7f448c13ff40 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4494055640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c140800 con 0x7f448c0acf60 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4494055640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c0aec00 con 0x7f448c1403f0 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4494055640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4494055640 10 monclient(hunting): authenticate will time out at 1120.506140s 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4491df1640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f448c0acf60 0x7f448c139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:35876/0 (socket says 172.21.15.59:35876) 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.391+0000 7f4491df1640 1 -- 172.21.15.59:0/3364359121 learned_addr learned my addr 172.21.15.59:0/3364359121 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.392+0000 7f4490def640 1 -- 172.21.15.59:0/3364359121 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640546458 0 0) 0x7f448c140800 con 0x7f448c0acf60 2024-03-29T11:24:30.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.392+0000 7f4490def640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.392+0000 7f4490def640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.392+0000 7f4490def640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.392+0000 7f4490def640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:24:30.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.392+0000 7f4490def640 1 -- 172.21.15.59:0/3364359121 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4478003520 con 0x7f448c0acf60 2024-03-29T11:24:30.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.393+0000 7f4490def640 1 -- 172.21.15.59:0/3364359121 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 953289523 0 0) 0x7f4478003520 con 0x7f448c0acf60 2024-03-29T11:24:30.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.394+0000 7f4490def640 1 -- 172.21.15.59:0/3364359121 >> v1:172.21.15.59:6790/0 conn(0x7f448c1403f0 legacy=0x7f448c14bf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:30.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.394+0000 7f4490def640 1 -- 172.21.15.59:0/3364359121 >> v1:172.21.15.59:6789/0 conn(0x7f448c13ff40 legacy=0x7f448c148840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:30.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 10 monclient: _finish_hunting 0 2024-03-29T11:24:30.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 1 monclient: found mon.noname-c 2024-03-29T11:24:30.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:30.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:24:30.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 1 -- 172.21.15.59:0/3364359121 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c0aeee0 con 0x7f448c0acf60 2024-03-29T11:24:30.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 10 monclient: _finish_auth 0 2024-03-29T11:24:30.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 10 monclient: _check_auth_tickets 2024-03-29T11:24:30.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:30.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4494055640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4494055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 1 -- 172.21.15.59:0/3364359121 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4488001690 con 0x7f448c0acf60 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.397+0000 7f4490def640 10 monclient: dump: 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:30.384 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.398+0000 7f4494055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.399+0000 7f4490def640 1 -- 172.21.15.59:0/3364359121 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4488001920 con 0x7f448c0acf60 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.399+0000 7f4490def640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.399+0000 7f4494055640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.399+0000 7f4494055640 4 set_mon_vals no callback set 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.399+0000 7f4490def640 1 -- 172.21.15.59:0/3364359121 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4488001cb0 con 0x7f448c0acf60 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.399+0000 7f4490def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.399+0000 7f4490def640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.399+0000 7f4490def640 10 monclient: dump: 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.385 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:30.386 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:30.386 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:30.386 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:30.386 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:30.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.400+0000 7f4494055640 10 monclient: shutdown 2024-03-29T11:24:30.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.400+0000 7f4494055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:30.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.400+0000 7f4494055640 1 -- 172.21.15.59:0/3364359121 >> v1:172.21.15.172:6789/0 conn(0x7f448c0acf60 legacy=0x7f448c139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.400+0000 7f4494055640 1 -- 172.21.15.59:0/3364359121 shutdown_connections 2024-03-29T11:24:30.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.400+0000 7f4494055640 1 -- 172.21.15.59:0/3364359121 >> 172.21.15.59:0/3364359121 conn(0x7f448c0b35e0 msgr2=0x7f448c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:30.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.400+0000 7f4494055640 1 -- 172.21.15.59:0/3364359121 shutdown_connections 2024-03-29T11:24:30.386 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.400+0000 7f4494055640 1 -- 172.21.15.59:0/3364359121 wait complete. 2024-03-29T11:24:30.387 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.401+0000 7f4494055640 10 monclient: build_initial_monmap 2024-03-29T11:24:30.387 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.401+0000 7f4494055640 10 monclient: monmap: 2024-03-29T11:24:30.387 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:30.387 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.401+0000 7f4494055640 1 librados: starting msgr at 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.401+0000 7f4494055640 1 librados: starting objecter 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.401+0000 7f4494055640 1 Processor -- start 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 1 -- start start 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 1 librados: setting wanted keys 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 1 librados: calling monclient init 2024-03-29T11:24:30.388 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 10 monclient: init 2024-03-29T11:24:30.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:30.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:30.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 10 monclient(hunting): picked mon.noname-c con 0x7f448c0acf60 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:30.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 10 monclient(hunting): picked mon.noname-a con 0x7f448c13ff40 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:30.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 10 monclient(hunting): picked mon.noname-b con 0x7f448c1403f0 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:30.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c072040 con 0x7f448c13ff40 2024-03-29T11:24:30.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c070b20 con 0x7f448c1403f0 2024-03-29T11:24:30.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c070dc0 con 0x7f448c0acf60 2024-03-29T11:24:30.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:30.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.402+0000 7f4494055640 10 monclient(hunting): authenticate will time out at 1120.517647s 2024-03-29T11:24:30.391 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.405+0000 7f44915f0640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f448c13ff40 0x7f448c1d4e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54781/0 (socket says 172.21.15.59:54781) 2024-03-29T11:24:30.391 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.405+0000 7f44915f0640 1 -- 172.21.15.59:0/52676208 learned_addr learned my addr 172.21.15.59:0/52676208 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:30.391 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.405+0000 7f44827fc640 10 client.?.objecter ms_handle_connect 0x7f448c13ff40 2024-03-29T11:24:30.391 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.405+0000 7f44827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:30.391 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.405+0000 7f44827fc640 10 client.?.objecter ms_handle_connect 0x7f448c1403f0 2024-03-29T11:24:30.391 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.405+0000 7f44827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:30.391 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.406+0000 7f44827fc640 10 client.?.objecter ms_handle_connect 0x7f448c0acf60 2024-03-29T11:24:30.391 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.406+0000 7f44827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:30.391 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.406+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291031417 0 0) 0x7f448c070dc0 con 0x7f448c0acf60 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4470004eb0 con 0x7f448c0acf60 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 163912469 0 0) 0x7f4470004eb0 con 0x7f448c0acf60 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f448c070dc0 con 0x7f448c0acf60 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4488001610 con 0x7f448c0acf60 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient(hunting): dump: 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:30.392 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2921627589 0 0) 0x7f448c070dc0 con 0x7f448c0acf60 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 >> v1:172.21.15.172:6789/0 conn(0x7f448c1403f0 legacy=0x7f448c1d5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 >> v1:172.21.15.59:6789/0 conn(0x7f448c13ff40 legacy=0x7f448c1d4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 1 monclient: found mon.c 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:30.393 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c0723f0 con 0x7f448c0acf60 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.407+0000 7f44827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4488004a90 con 0x7f448c0acf60 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f44827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4488004da0 con 0x7f448c0acf60 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f44827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f44827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f44827fc640 10 monclient: dump: 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:30.394 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f4494055640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f4494055640 10 monclient: _renew_subs 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f4494055640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f4494055640 1 -- 172.21.15.59:0/52676208 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448c0725e0 con 0x7f448c0acf60 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f4494055640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f4494055640 10 monclient: _renew_subs 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f4494055640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f4494055640 1 -- 172.21.15.59:0/52676208 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f448c0703c0 con 0x7f448c0acf60 2024-03-29T11:24:30.395 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.408+0000 7f4494055640 1 librados: init done 2024-03-29T11:24:30.396 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.410+0000 7f44837fe640 4 set_mon_vals no callback set 2024-03-29T11:24:30.396 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.410+0000 7f4494055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.396 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.411+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f448801ad70 con 0x7f448c0acf60 2024-03-29T11:24:30.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.411+0000 7f4494055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.411+0000 7f4494055640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:30.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.411+0000 7f4494055640 1 -- 172.21.15.59:0/52676208 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4474005080 con 0x7f448c0acf60 2024-03-29T11:24:30.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.412+0000 7f44915f0640 10 monclient: get_auth_request con 0x7f447006dc90 auth_method 0 2024-03-29T11:24:30.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.412+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f4488054870 con 0x7f448c0acf60 2024-03-29T11:24:30.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.412+0000 7f44827fc640 10 client.4403.objecter ms_dispatch 0x7f448c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:30.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.412+0000 7f44827fc640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:30.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.412+0000 7f44827fc640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:30.397 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.412+0000 7f44827fc640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T11:24:30.399 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.415+0000 7f44827fc640 10 client.4403.objecter ms_handle_connect 0x7f447006dc90 2024-03-29T11:24:30.403 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.420+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4488050b10 con 0x7f448c0acf60 2024-03-29T11:24:30.403 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.420+0000 7f44827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.403 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.420+0000 7f44827fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:30.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.764+0000 7f4494055640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:30.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.764+0000 7f4494055640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:30.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.765+0000 7f4494055640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:30.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.765+0000 7f4494055640 1 -- 172.21.15.59:0/52676208 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4474005330 con 0x7f448c0acf60 2024-03-29T11:24:30.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.765+0000 7f44827fc640 1 -- 172.21.15.59:0/52676208 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f4488050cb0 con 0x7f448c0acf60 2024-03-29T11:24:30.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.765+0000 7f44827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:30.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.765+0000 7f44827fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:30.748 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:24:30.748 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:30.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 10 librados: watch_flush enter 2024-03-29T11:24:30.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 10 librados: watch_flush exit 2024-03-29T11:24:30.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T11:24:30.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 10 client.4403.objecter successfully canceled tick 2024-03-29T11:24:30.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 1 -- 172.21.15.59:0/52676208 >> v1:172.21.15.172:6816/2420742773 conn(0x7f447006dc90 legacy=0x7f4470070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 10 monclient: shutdown 2024-03-29T11:24:30.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:30.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 1 -- 172.21.15.59:0/52676208 >> v1:172.21.15.59:6790/0 conn(0x7f448c0acf60 legacy=0x7f448c1346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 1 -- 172.21.15.59:0/52676208 shutdown_connections 2024-03-29T11:24:30.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 1 -- 172.21.15.59:0/52676208 >> 172.21.15.59:0/52676208 conn(0x7f448c0b35e0 msgr2=0x7f448c0a9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:30.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 1 -- 172.21.15.59:0/52676208 shutdown_connections 2024-03-29T11:24:30.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 1 -- 172.21.15.59:0/52676208 wait complete. 2024-03-29T11:24:30.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:30.769+0000 7f4494055640 1 librados: shutdown 2024-03-29T11:24:30.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:37.577 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:37.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.752+0000 7fb87c9e8640 10 monclient: get_monmap_and_config 2024-03-29T11:24:37.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.752+0000 7fb87c9e8640 10 monclient: build_initial_monmap 2024-03-29T11:24:37.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.752+0000 7fb87c9e8640 10 monclient: monmap: 2024-03-29T11:24:37.735 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:37.735 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:37.735 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:24:37.753807+0000 2024-03-29T11:24:37.736 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:24:37.753807+0000 2024-03-29T11:24:37.736 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:37.736 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:37.736 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:37.736 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:24:37.736 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:24:37.736 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:37.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.752+0000 7fb87c9e8640 1 Processor -- start 2024-03-29T11:24:37.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.753+0000 7fb87c9e8640 1 -- start start 2024-03-29T11:24:37.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.753+0000 7fb87c9e8640 10 monclient: init 2024-03-29T11:24:37.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.754+0000 7fb87c9e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:37.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.754+0000 7fb87c9e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:37.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.754+0000 7fb87c9e8640 10 monclient(hunting): picked mon.noname-c con 0x7fb8780a9930 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:37.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.754+0000 7fb87c9e8640 10 monclient(hunting): picked mon.noname-b con 0x7fb878149dc0 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:37.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.754+0000 7fb87c9e8640 10 monclient(hunting): picked mon.noname-a con 0x7fb87814ff40 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:37.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.754+0000 7fb87c9e8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780aae60 con 0x7fb87814ff40 2024-03-29T11:24:37.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.754+0000 7fb87c9e8640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780b4310 con 0x7fb8780a9930 2024-03-29T11:24:37.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.754+0000 7fb87c9e8640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780afdd0 con 0x7fb878149dc0 2024-03-29T11:24:37.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.754+0000 7fb87c9e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:37.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.754+0000 7fb87c9e8640 10 monclient(hunting): authenticate will time out at 1127.868857s 2024-03-29T11:24:37.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.760+0000 7fb87659c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb8780a9930 0x7fb878143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:46897/0 (socket says 172.21.15.59:46897) 2024-03-29T11:24:37.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.760+0000 7fb87659c640 1 -- 172.21.15.59:0/3996397955 learned_addr learned my addr 172.21.15.59:0/3996397955 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:37.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282253223 0 0) 0x7fb8780b4310 con 0x7fb8780a9930 2024-03-29T11:24:37.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:24:37.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb860003520 con 0x7fb8780a9930 2024-03-29T11:24:37.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871329318 0 0) 0x7fb8780aae60 con 0x7fb87814ff40 2024-03-29T11:24:37.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8780b4310 con 0x7fb87814ff40 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302196517 0 0) 0x7fb8780afdd0 con 0x7fb878149dc0 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.762+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8780aae60 con 0x7fb878149dc0 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4256977777 0 0) 0x7fb8780b4310 con 0x7fb87814ff40 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 >> v1:172.21.15.59:6790/0 conn(0x7fb878149dc0 legacy=0x7fb87814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 >> v1:172.21.15.172:6789/0 conn(0x7fb8780a9930 legacy=0x7fb878143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 1 monclient: found mon.noname-a 2024-03-29T11:24:37.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87814a950 con 0x7fb87814ff40 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 10 monclient: _finish_auth 0 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87c9e8640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87c9e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb870001920 con 0x7fb87814ff40 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 10 monclient: dump: 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:37.748 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:37.749 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.749 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:37.749 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:37.749 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:37.749 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:37.749 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:37.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87c9e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:37.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb870001b40 con 0x7fb87814ff40 2024-03-29T11:24:37.749 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87c9e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.763+0000 7fb87c9e8640 4 set_mon_vals no callback set 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.764+0000 7fb87559a640 1 -- 172.21.15.59:0/3996397955 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb870001ed0 con 0x7fb87814ff40 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.764+0000 7fb87559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.764+0000 7fb87559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.764+0000 7fb87559a640 10 monclient: dump: 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:37.751 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:37.752 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:37.752 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:37.752 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:37.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.764+0000 7fb87c9e8640 10 monclient: shutdown 2024-03-29T11:24:37.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.764+0000 7fb87c9e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:37.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.764+0000 7fb87c9e8640 1 -- 172.21.15.59:0/3996397955 >> v1:172.21.15.59:6789/0 conn(0x7fb87814ff40 legacy=0x7fb878152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.767+0000 7fb87c9e8640 1 -- 172.21.15.59:0/3996397955 shutdown_connections 2024-03-29T11:24:37.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.767+0000 7fb87c9e8640 1 -- 172.21.15.59:0/3996397955 >> 172.21.15.59:0/3996397955 conn(0x7fb8780b35e0 msgr2=0x7fb8780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:37.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.767+0000 7fb87c9e8640 1 -- 172.21.15.59:0/3996397955 shutdown_connections 2024-03-29T11:24:37.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.768+0000 7fb87c9e8640 1 -- 172.21.15.59:0/3996397955 wait complete. 2024-03-29T11:24:37.756 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.769+0000 7fb87c9e8640 10 monclient: build_initial_monmap 2024-03-29T11:24:37.756 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.769+0000 7fb87c9e8640 10 monclient: monmap: 2024-03-29T11:24:37.756 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:37.756 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:37.756 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:24:37.756 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:24:37.756 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:37.757 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:37.757 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:37.757 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:24:37.757 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:24:37.757 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:37.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.769+0000 7fb87c9e8640 1 librados: starting msgr at 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.769+0000 7fb87c9e8640 1 librados: starting objecter 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.769+0000 7fb87c9e8640 1 Processor -- start 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.769+0000 7fb87c9e8640 1 -- start start 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.769+0000 7fb87c9e8640 1 librados: setting wanted keys 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.769+0000 7fb87c9e8640 1 librados: calling monclient init 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.769+0000 7fb87c9e8640 10 monclient: init 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.770+0000 7fb87c9e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.770+0000 7fb87c9e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.770+0000 7fb87c9e8640 10 monclient(hunting): picked mon.noname-b con 0x7fb8780a9930 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.770+0000 7fb87c9e8640 10 monclient(hunting): picked mon.noname-a con 0x7fb878149dc0 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.770+0000 7fb87c9e8640 10 monclient(hunting): picked mon.noname-c con 0x7fb87814ff40 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.770+0000 7fb87c9e8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb878144640 con 0x7fb878149dc0 2024-03-29T11:24:37.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.770+0000 7fb87c9e8640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87813bf70 con 0x7fb8780a9930 2024-03-29T11:24:37.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.770+0000 7fb87c9e8640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8781471f0 con 0x7fb87814ff40 2024-03-29T11:24:37.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.770+0000 7fb87c9e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:37.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.770+0000 7fb87c9e8640 10 monclient(hunting): authenticate will time out at 1127.885236s 2024-03-29T11:24:37.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.775+0000 7fb87659c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb8780a9930 0x7fb8780ac220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:46987/0 (socket says 172.21.15.59:46987) 2024-03-29T11:24:37.767 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.775+0000 7fb87659c640 1 -- 172.21.15.59:0/730322768 learned_addr learned my addr 172.21.15.59:0/730322768 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.778+0000 7fb866ffd640 10 client.?.objecter ms_handle_connect 0x7fb87814ff40 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.778+0000 7fb866ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.778+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629979503 0 0) 0x7fb8781471f0 con 0x7fb87814ff40 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.778+0000 7fb866ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.778+0000 7fb866ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.778+0000 7fb866ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.778+0000 7fb866ffd640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.778+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb858004eb0 con 0x7fb87814ff40 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 client.?.objecter ms_handle_connect 0x7fb8780a9930 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 294873131 0 0) 0x7fb858004eb0 con 0x7fb87814ff40 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8781471f0 con 0x7fb87814ff40 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 client.?.objecter ms_handle_connect 0x7fb878149dc0 2024-03-29T11:24:37.768 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb870001e50 con 0x7fb87814ff40 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:37.769 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 938730067 0 0) 0x7fb8781471f0 con 0x7fb87814ff40 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 >> v1:172.21.15.172:6789/0 conn(0x7fb8780a9930 legacy=0x7fb8780ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 >> v1:172.21.15.59:6789/0 conn(0x7fb878149dc0 legacy=0x7fb878144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 1 monclient: found mon.c 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8781474d0 con 0x7fb87814ff40 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb87c9e8640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb87c9e8640 10 monclient: _renew_subs 2024-03-29T11:24:37.770 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb87c9e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb87c9e8640 1 -- 172.21.15.59:0/730322768 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87806e840 con 0x7fb87814ff40 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb87c9e8640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb87c9e8640 10 monclient: _renew_subs 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb87c9e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb87c9e8640 1 -- 172.21.15.59:0/730322768 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87813dc50 con 0x7fb87814ff40 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb87c9e8640 1 librados: init done 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb870001fb0 con 0x7fb87814ff40 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.779+0000 7fb866ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.780+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb870003dc0 con 0x7fb87814ff40 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.780+0000 7fb866ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.780+0000 7fb866ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.780+0000 7fb866ffd640 10 monclient: dump: 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:37.771 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.781+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fb870019d90 con 0x7fb87814ff40 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.782+0000 7fb874d99640 4 set_mon_vals no callback set 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.782+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fb870083ba0 con 0x7fb87814ff40 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.782+0000 7fb866ffd640 10 client.4409.objecter ms_dispatch 0x7fb878000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.782+0000 7fb866ffd640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.782+0000 7fb866ffd640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:37.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.782+0000 7fb866ffd640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.782+0000 7fb87659c640 10 monclient: get_auth_request con 0x7fb85806dd50 auth_method 0 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.783+0000 7fb866ffd640 10 client.4409.objecter ms_handle_connect 0x7fb85806dd50 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.783+0000 7fb87c9e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.783+0000 7fb87c9e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.783+0000 7fb87c9e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:37.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.783+0000 7fb87c9e8640 1 -- 172.21.15.59:0/730322768 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c005080 con 0x7fb87814ff40 2024-03-29T11:24:37.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.794+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb87004f2b0 con 0x7fb87814ff40 2024-03-29T11:24:37.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.794+0000 7fb866ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:37.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:37.794+0000 7fb866ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:38.106 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.123+0000 7fb87c9e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.106 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.123+0000 7fb87c9e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.107 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.123+0000 7fb87c9e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:24:38.107 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.123+0000 7fb87c9e8640 1 -- 172.21.15.59:0/730322768 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb85c005330 con 0x7fb87814ff40 2024-03-29T11:24:38.107 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.124+0000 7fb866ffd640 1 -- 172.21.15.59:0/730322768 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fb8700513b0 con 0x7fb87814ff40 2024-03-29T11:24:38.107 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.124+0000 7fb866ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.108 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.124+0000 7fb866ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:38.108 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:24:38.108 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:38.111 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.128+0000 7fb87c9e8640 10 librados: watch_flush enter 2024-03-29T11:24:38.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.129+0000 7fb87c9e8640 10 librados: watch_flush exit 2024-03-29T11:24:38.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.129+0000 7fb87c9e8640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:24:38.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.129+0000 7fb87c9e8640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:24:38.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.130+0000 7fb87c9e8640 1 -- 172.21.15.59:0/730322768 >> v1:172.21.15.172:6816/2420742773 conn(0x7fb85806dd50 legacy=0x7fb858070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.130+0000 7fb87c9e8640 10 monclient: shutdown 2024-03-29T11:24:38.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.130+0000 7fb87c9e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:38.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.130+0000 7fb87c9e8640 1 -- 172.21.15.59:0/730322768 >> v1:172.21.15.59:6790/0 conn(0x7fb87814ff40 legacy=0x7fb8781451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.130+0000 7fb87c9e8640 1 -- 172.21.15.59:0/730322768 shutdown_connections 2024-03-29T11:24:38.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.130+0000 7fb87c9e8640 1 -- 172.21.15.59:0/730322768 >> 172.21.15.59:0/730322768 conn(0x7fb8780b35e0 msgr2=0x7fb8780b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:38.116 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.133+0000 7fb87c9e8640 1 -- 172.21.15.59:0/730322768 shutdown_connections 2024-03-29T11:24:38.116 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.133+0000 7fb87c9e8640 1 -- 172.21.15.59:0/730322768 wait complete. 2024-03-29T11:24:38.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:38.136+0000 7fb87c9e8640 1 librados: shutdown 2024-03-29T11:24:38.137 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:44.944 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.108+0000 7f2b537f4640 10 monclient: get_monmap_and_config 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.108+0000 7f2b537f4640 10 monclient: build_initial_monmap 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.108+0000 7f2b537f4640 10 monclient: monmap: 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:24:45.109177+0000 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:24:45.109177+0000 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:24:45.091 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:24:45.092 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:45.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.109+0000 7f2b537f4640 1 Processor -- start 2024-03-29T11:24:45.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.109+0000 7f2b537f4640 1 -- start start 2024-03-29T11:24:45.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.109+0000 7f2b537f4640 10 monclient: init 2024-03-29T11:24:45.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.110+0000 7f2b537f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:45.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.110+0000 7f2b537f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:45.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.110+0000 7f2b537f4640 10 monclient(hunting): picked mon.noname-a con 0x7f2b4c0b0770 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:45.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.110+0000 7f2b537f4640 10 monclient(hunting): picked mon.noname-b con 0x7f2b4c14a700 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:45.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.110+0000 7f2b537f4640 10 monclient(hunting): picked mon.noname-c con 0x7f2b4c0ac5a0 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:45.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.110+0000 7f2b537f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c0aba70 con 0x7f2b4c0b0770 2024-03-29T11:24:45.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.110+0000 7f2b537f4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c0aecb0 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.110+0000 7f2b537f4640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c0af150 con 0x7f2b4c14a700 2024-03-29T11:24:45.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.110+0000 7f2b537f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:45.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.110+0000 7f2b537f4640 10 monclient(hunting): authenticate will time out at 1135.224276s 2024-03-29T11:24:45.096 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.113+0000 7f2b51590640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2b4c0b0770 0x7f2b4c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44871/0 (socket says 172.21.15.59:44871) 2024-03-29T11:24:45.096 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.113+0000 7f2b51590640 1 -- 172.21.15.59:0/1631516454 learned_addr learned my addr 172.21.15.59:0/1631516454 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:45.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.114+0000 7f2b43fff640 1 -- 172.21.15.59:0/1631516454 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265559149 0 0) 0x7f2b4c0aecb0 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.114+0000 7f2b43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.114+0000 7f2b43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.114+0000 7f2b43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.114+0000 7f2b43fff640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:24:45.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.114+0000 7f2b43fff640 1 -- 172.21.15.59:0/1631516454 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b38003520 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.115+0000 7f2b43fff640 1 -- 172.21.15.59:0/1631516454 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2626504277 0 0) 0x7f2b38003520 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.115+0000 7f2b43fff640 1 -- 172.21.15.59:0/1631516454 >> v1:172.21.15.59:6790/0 conn(0x7f2b4c14a700 legacy=0x7f2b4c0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:45.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.117+0000 7f2b43fff640 1 -- 172.21.15.59:0/1631516454 >> v1:172.21.15.59:6789/0 conn(0x7f2b4c0b0770 legacy=0x7f2b4c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:45.102 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:45.102 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 1 monclient: found mon.noname-c 2024-03-29T11:24:45.102 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:45.102 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:24:45.102 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 1 -- 172.21.15.59:0/1631516454 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c0ac9d0 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.102 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:45.102 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:45.102 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:45.102 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 1 -- 172.21.15.59:0/1631516454 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2b480016b0 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.102 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b43fff640 10 monclient: dump: 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b537f4640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:24:45.103 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.118+0000 7f2b537f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.119+0000 7f2b43fff640 1 -- 172.21.15.59:0/1631516454 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b48001910 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.119+0000 7f2b43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.119+0000 7f2b43fff640 1 -- 172.21.15.59:0/1631516454 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2b48001d60 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.119+0000 7f2b43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.119+0000 7f2b43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.119+0000 7f2b43fff640 10 monclient: dump: 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:45.104 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:45.105 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:45.105 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.119+0000 7f2b537f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:45.105 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.119+0000 7f2b537f4640 4 set_mon_vals no callback set 2024-03-29T11:24:45.105 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.120+0000 7f2b537f4640 10 monclient: shutdown 2024-03-29T11:24:45.106 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.120+0000 7f2b537f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:45.106 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.120+0000 7f2b537f4640 1 -- 172.21.15.59:0/1631516454 >> v1:172.21.15.172:6789/0 conn(0x7f2b4c0ac5a0 legacy=0x7f2b4c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.106 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.122+0000 7f2b537f4640 1 -- 172.21.15.59:0/1631516454 shutdown_connections 2024-03-29T11:24:45.106 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.122+0000 7f2b537f4640 1 -- 172.21.15.59:0/1631516454 >> 172.21.15.59:0/1631516454 conn(0x7f2b4c06b660 msgr2=0x7f2b4c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:45.106 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.123+0000 7f2b537f4640 1 -- 172.21.15.59:0/1631516454 shutdown_connections 2024-03-29T11:24:45.106 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.123+0000 7f2b537f4640 1 -- 172.21.15.59:0/1631516454 wait complete. 2024-03-29T11:24:45.111 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.124+0000 7f2b537f4640 10 monclient: build_initial_monmap 2024-03-29T11:24:45.111 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.124+0000 7f2b537f4640 10 monclient: monmap: 2024-03-29T11:24:45.111 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:45.111 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:45.111 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:24:45.111 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:24:45.111 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:45.111 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:45.111 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:45.111 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:24:45.111 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.124+0000 7f2b537f4640 1 librados: starting msgr at 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.124+0000 7f2b537f4640 1 librados: starting objecter 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.124+0000 7f2b537f4640 1 Processor -- start 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.124+0000 7f2b537f4640 1 -- start start 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.124+0000 7f2b537f4640 1 librados: setting wanted keys 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.124+0000 7f2b537f4640 1 librados: calling monclient init 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.124+0000 7f2b537f4640 10 monclient: init 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.125+0000 7f2b537f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.125+0000 7f2b537f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.127+0000 7f2b537f4640 10 monclient(hunting): picked mon.noname-a con 0x7f2b4c0ac5a0 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.127+0000 7f2b537f4640 10 monclient(hunting): picked mon.noname-c con 0x7f2b4c0b0770 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.127+0000 7f2b537f4640 10 monclient(hunting): picked mon.noname-b con 0x7f2b4c14a700 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:45.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.127+0000 7f2b537f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c1e3f30 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.127+0000 7f2b537f4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c11fc50 con 0x7f2b4c14a700 2024-03-29T11:24:45.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.127+0000 7f2b537f4640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c11eaa0 con 0x7f2b4c0b0770 2024-03-29T11:24:45.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.127+0000 7f2b537f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:45.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.127+0000 7f2b537f4640 10 monclient(hunting): authenticate will time out at 1135.241209s 2024-03-29T11:24:45.120 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.133+0000 7f2b51590640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2b4c0ac5a0 0x7f2b4c11e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:45047/0 (socket says 172.21.15.59:45047) 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.133+0000 7f2b51590640 1 -- 172.21.15.59:0/2566615181 learned_addr learned my addr 172.21.15.59:0/2566615181 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 client.?.objecter ms_handle_connect 0x7f2b4c0ac5a0 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 client.?.objecter ms_handle_connect 0x7f2b4c0b0770 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781880429 0 0) 0x7f2b4c1e3f30 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b30004eb0 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424427677 0 0) 0x7f2b4c11eaa0 con 0x7f2b4c0b0770 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b4c1e3f30 con 0x7f2b4c0b0770 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.134+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4232328115 0 0) 0x7f2b4c1e3f30 con 0x7f2b4c0b0770 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b4c11eaa0 con 0x7f2b4c0b0770 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1386656942 0 0) 0x7f2b30004eb0 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b4c1e3f30 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2b3c001790 con 0x7f2b4c0b0770 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:45.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 10 monclient(hunting): dump: 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2b44001cb0 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.123 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 10 monclient(hunting): dump: 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 10 client.?.objecter ms_handle_connect 0x7f2b4c14a700 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.135+0000 7f2b41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:45.124 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 789133369 0 0) 0x7f2b4c1e3f30 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 >> v1:172.21.15.59:6790/0 conn(0x7f2b4c0b0770 legacy=0x7f2b4c1e4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 >> v1:172.21.15.172:6789/0 conn(0x7f2b4c14a700 legacy=0x7f2b4c1e0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b41ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b41ffb640 1 monclient: found mon.a 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b41ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c11ec30 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b41ffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b41ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b537f4640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b537f4640 10 monclient: _renew_subs 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b537f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:45.125 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.136+0000 7f2b537f4640 1 -- 172.21.15.59:0/2566615181 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4c11b1c0 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b440020c0 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2b44004d40 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b41ffb640 10 monclient: dump: 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:45.126 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:45.127 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:45.127 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:45.127 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:45.127 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b537f4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:45.127 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b537f4640 10 monclient: _renew_subs 2024-03-29T11:24:45.127 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b537f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:45.127 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b537f4640 1 -- 172.21.15.59:0/2566615181 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b4c1e4d00 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.127 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b537f4640 1 librados: init done 2024-03-29T11:24:45.127 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.137+0000 7f2b437fe640 4 set_mon_vals no callback set 2024-03-29T11:24:45.127 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.139+0000 7f2b537f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.139+0000 7f2b537f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.139+0000 7f2b537f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:45.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.139+0000 7f2b537f4640 1 -- 172.21.15.59:0/2566615181 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4c0ab560 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.128 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.142+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f2b44004ea0 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.143+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f2b44084d50 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.143+0000 7f2b41ffb640 10 client.4416.objecter ms_dispatch 0x7f2b4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:45.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.143+0000 7f2b41ffb640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:45.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.143+0000 7f2b41ffb640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:45.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.143+0000 7f2b41ffb640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T11:24:45.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.143+0000 7f2b50d8f640 10 monclient: get_auth_request con 0x7f2b3006dec0 auth_method 0 2024-03-29T11:24:45.129 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.144+0000 7f2b41ffb640 10 client.4416.objecter ms_handle_connect 0x7f2b3006dec0 2024-03-29T11:24:45.134 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.152+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b44050630 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.134 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.152+0000 7f2b41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.134 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.152+0000 7f2b41ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:45.463 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.480+0000 7f2b537f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.463 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.480+0000 7f2b537f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.463 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.480+0000 7f2b537f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:45.463 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.480+0000 7f2b537f4640 1 -- 172.21.15.59:0/2566615181 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b4c075f10 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.463 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.481+0000 7f2b41ffb640 1 -- 172.21.15.59:0/2566615181 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f2b44052560 con 0x7f2b4c0ac5a0 2024-03-29T11:24:45.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.482+0000 7f2b41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.482+0000 7f2b41ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:45.467 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:24:45.467 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:45.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.485+0000 7f2b537f4640 10 librados: watch_flush enter 2024-03-29T11:24:45.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.485+0000 7f2b537f4640 10 librados: watch_flush exit 2024-03-29T11:24:45.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.485+0000 7f2b537f4640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T11:24:45.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.485+0000 7f2b537f4640 10 client.4416.objecter successfully canceled tick 2024-03-29T11:24:45.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.486+0000 7f2b537f4640 1 -- 172.21.15.59:0/2566615181 >> v1:172.21.15.172:6816/2420742773 conn(0x7f2b3006dec0 legacy=0x7f2b30070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.486+0000 7f2b537f4640 10 monclient: shutdown 2024-03-29T11:24:45.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.486+0000 7f2b537f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:45.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.486+0000 7f2b537f4640 1 -- 172.21.15.59:0/2566615181 >> v1:172.21.15.59:6789/0 conn(0x7f2b4c0ac5a0 legacy=0x7f2b4c11e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.486+0000 7f2b537f4640 1 -- 172.21.15.59:0/2566615181 shutdown_connections 2024-03-29T11:24:45.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.486+0000 7f2b537f4640 1 -- 172.21.15.59:0/2566615181 >> 172.21.15.59:0/2566615181 conn(0x7f2b4c06b660 msgr2=0x7f2b4c0abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:45.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.486+0000 7f2b537f4640 1 -- 172.21.15.59:0/2566615181 shutdown_connections 2024-03-29T11:24:45.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.486+0000 7f2b537f4640 1 -- 172.21.15.59:0/2566615181 wait complete. 2024-03-29T11:24:45.469 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:45.486+0000 7f2b537f4640 1 librados: shutdown 2024-03-29T11:24:45.486 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:52.295 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:52.446 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.463+0000 7f7c8080f640 10 monclient: get_monmap_and_config 2024-03-29T11:24:52.446 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.463+0000 7f7c8080f640 10 monclient: build_initial_monmap 2024-03-29T11:24:52.446 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.463+0000 7f7c8080f640 10 monclient: monmap: 2024-03-29T11:24:52.446 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:52.446 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:52.447 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:24:52.464630+0000 2024-03-29T11:24:52.447 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:24:52.464630+0000 2024-03-29T11:24:52.447 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:52.447 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:52.447 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:52.447 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:24:52.447 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:24:52.447 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:52.450 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.467+0000 7f7c8080f640 1 Processor -- start 2024-03-29T11:24:52.450 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.468+0000 7f7c8080f640 1 -- start start 2024-03-29T11:24:52.450 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.468+0000 7f7c8080f640 10 monclient: init 2024-03-29T11:24:52.451 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.469+0000 7f7c8080f640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:52.452 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.469+0000 7f7c8080f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:52.452 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.469+0000 7f7c8080f640 10 monclient(hunting): picked mon.noname-c con 0x7f7c7c1457f0 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:52.452 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.469+0000 7f7c8080f640 10 monclient(hunting): picked mon.noname-b con 0x7f7c7c13f650 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:52.452 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.469+0000 7f7c8080f640 10 monclient(hunting): picked mon.noname-a con 0x7f7c7c0a9d70 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:52.452 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.469+0000 7f7c8080f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c0aaef0 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.452 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.469+0000 7f7c8080f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c145bd0 con 0x7f7c7c1457f0 2024-03-29T11:24:52.452 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.469+0000 7f7c8080f640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c145eb0 con 0x7f7c7c13f650 2024-03-29T11:24:52.453 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.469+0000 7f7c8080f640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:52.453 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.469+0000 7f7c8080f640 10 monclient(hunting): authenticate will time out at 1142.583101s 2024-03-29T11:24:52.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.474+0000 7f7c7ad9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7c7c0a9d70 0x7f7c7c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:45967/0 (socket says 172.21.15.59:45967) 2024-03-29T11:24:52.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.474+0000 7f7c7ad9d640 1 -- 172.21.15.59:0/160822816 learned_addr learned my addr 172.21.15.59:0/160822816 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:52.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320497040 0 0) 0x7f7c7c0aaef0 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.458 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.459 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:24:52.459 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c64003520 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.463 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452478303 0 0) 0x7f7c7c145bd0 con 0x7f7c7c1457f0 2024-03-29T11:24:52.463 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.463 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.463 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.476+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c7c0aaef0 con 0x7f7c7c1457f0 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733298218 0 0) 0x7f7c7c145eb0 con 0x7f7c7c13f650 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c7c145bd0 con 0x7f7c7c13f650 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 634934702 0 0) 0x7f7c64003520 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 >> v1:172.21.15.59:6790/0 conn(0x7f7c7c13f650 legacy=0x7f7c7c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 >> v1:172.21.15.172:6789/0 conn(0x7f7c7c1457f0 legacy=0x7f7c7c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 1 monclient: found mon.noname-a 2024-03-29T11:24:52.464 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c146340 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 10 monclient: _finish_auth 0 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.477+0000 7f7c7959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.478+0000 7f7c8080f640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.478+0000 7f7c8080f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.478+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7c70001690 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.478+0000 7f7c7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.478+0000 7f7c7959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.478+0000 7f7c7959a640 10 monclient: dump: 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:52.465 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.478+0000 7f7c8080f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.478+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c70001870 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.478+0000 7f7c7959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.478+0000 7f7c8080f640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.478+0000 7f7c8080f640 4 set_mon_vals no callback set 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.479+0000 7f7c7959a640 1 -- 172.21.15.59:0/160822816 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7c70001690 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.466 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.479+0000 7f7c7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.479+0000 7f7c7959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.479+0000 7f7c7959a640 10 monclient: dump: 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.479+0000 7f7c8080f640 10 monclient: shutdown 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.479+0000 7f7c8080f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:52.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.479+0000 7f7c8080f640 1 -- 172.21.15.59:0/160822816 >> v1:172.21.15.59:6789/0 conn(0x7f7c7c0a9d70 legacy=0x7f7c7c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.480+0000 7f7c8080f640 1 -- 172.21.15.59:0/160822816 shutdown_connections 2024-03-29T11:24:52.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.480+0000 7f7c8080f640 1 -- 172.21.15.59:0/160822816 >> 172.21.15.59:0/160822816 conn(0x7f7c7c066a00 msgr2=0x7f7c7c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:52.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.480+0000 7f7c8080f640 1 -- 172.21.15.59:0/160822816 shutdown_connections 2024-03-29T11:24:52.468 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.480+0000 7f7c8080f640 1 -- 172.21.15.59:0/160822816 wait complete. 2024-03-29T11:24:52.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.482+0000 7f7c8080f640 10 monclient: build_initial_monmap 2024-03-29T11:24:52.470 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.482+0000 7f7c8080f640 10 monclient: monmap: 2024-03-29T11:24:52.470 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:52.470 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:52.470 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:24:52.470 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:24:52.470 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:52.470 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:52.470 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.482+0000 7f7c8080f640 1 librados: starting msgr at 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.482+0000 7f7c8080f640 1 librados: starting objecter 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.482+0000 7f7c8080f640 1 Processor -- start 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.482+0000 7f7c8080f640 1 -- start start 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 1 librados: setting wanted keys 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 1 librados: calling monclient init 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 10 monclient: init 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 10 monclient(hunting): picked mon.noname-b con 0x7f7c7c0a9d70 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 10 monclient(hunting): picked mon.noname-c con 0x7f7c7c13f650 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:52.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 10 monclient(hunting): picked mon.noname-a con 0x7f7c7c1457f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:52.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c1422e0 con 0x7f7c7c1457f0 2024-03-29T11:24:52.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c142060 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c0ac230 con 0x7f7c7c13f650 2024-03-29T11:24:52.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:52.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.483+0000 7f7c8080f640 10 monclient(hunting): authenticate will time out at 1142.597572s 2024-03-29T11:24:52.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.487+0000 7f7c7ad9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7c7c1457f0 0x7f7c7c1e2d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46235/0 (socket says 172.21.15.59:46235) 2024-03-29T11:24:52.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.487+0000 7f7c7ad9d640 1 -- 172.21.15.59:0/2552889694 learned_addr learned my addr 172.21.15.59:0/2552889694 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:52.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.487+0000 7f7c6affd640 10 client.?.objecter ms_handle_connect 0x7f7c7c1457f0 2024-03-29T11:24:52.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.488+0000 7f7c6affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.488+0000 7f7c6affd640 10 client.?.objecter ms_handle_connect 0x7f7c7c13f650 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.488+0000 7f7c6affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.489+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146886231 0 0) 0x7f7c7c0ac230 con 0x7f7c7c13f650 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.489+0000 7f7c6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.489+0000 7f7c6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.489+0000 7f7c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.489+0000 7f7c6affd640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.489+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c5c004eb0 con 0x7f7c7c13f650 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.490+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884428709 0 0) 0x7f7c7c1422e0 con 0x7f7c7c1457f0 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.490+0000 7f7c6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c7c0ac230 con 0x7f7c7c1457f0 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 client.?.objecter ms_handle_connect 0x7f7c7c0a9d70 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3871000518 0 0) 0x7f7c5c004eb0 con 0x7f7c7c13f650 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c7c1422e0 con 0x7f7c7c13f650 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7c74001ba0 con 0x7f7c7c13f650 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): dump: 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.475 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959655889 0 0) 0x7f7c7c142060 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:24:52.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c5c004eb0 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 776028257 0 0) 0x7f7c7c0ac230 con 0x7f7c7c1457f0 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c7c142060 con 0x7f7c7c1457f0 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7c70001690 con 0x7f7c7c1457f0 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 10 monclient(hunting): dump: 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:52.477 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:52.478 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:52.480 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.491+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1353495307 0 0) 0x7f7c5c004eb0 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.480 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c7c0ac230 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.480 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7c6c001850 con 0x7f7c7c0a9d70 2024-03-29T11:24:52.480 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.480 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:52.480 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 10 monclient(hunting): dump: 2024-03-29T11:24:52.480 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:52.480 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:52.480 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.480 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3299291325 0 0) 0x7f7c7c142060 con 0x7f7c7c1457f0 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 >> v1:172.21.15.59:6790/0 conn(0x7f7c7c13f650 legacy=0x7f7c7c1370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 >> v1:172.21.15.172:6789/0 conn(0x7f7c7c0a9d70 legacy=0x7f7c7c142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 1 monclient: found mon.a 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c0ac510 con 0x7f7c7c1457f0 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 10 monclient: _finish_auth 0 2024-03-29T11:24:52.481 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c8080f640 5 monclient: authenticate success, global_id 4422 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c8080f640 10 monclient: _renew_subs 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c8080f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c8080f640 1 -- 172.21.15.59:0/2552889694 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7c0ac7f0 con 0x7f7c7c1457f0 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c8080f640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c8080f640 10 monclient: _renew_subs 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c8080f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c8080f640 1 -- 172.21.15.59:0/2552889694 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7c144a60 con 0x7f7c7c1457f0 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.492+0000 7f7c8080f640 1 librados: init done 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.494+0000 7f7c8080f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.494+0000 7f7c8080f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.494+0000 7f7c8080f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:52.482 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.494+0000 7f7c8080f640 1 -- 172.21.15.59:0/2552889694 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7c075f10 con 0x7f7c7c1457f0 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.495+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c70001200 con 0x7f7c7c1457f0 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.495+0000 7f7c6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.495+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f7c70004b40 con 0x7f7c7c1457f0 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.495+0000 7f7c78d99640 4 set_mon_vals no callback set 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.495+0000 7f7c6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.495+0000 7f7c6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.495+0000 7f7c6affd640 10 monclient: dump: 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:52.483 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:52.484 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:52.484 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:52.484 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.495+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f7c7001ab10 con 0x7f7c7c1457f0 2024-03-29T11:24:52.484 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.496+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f7c70084b40 con 0x7f7c7c1457f0 2024-03-29T11:24:52.484 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.496+0000 7f7c6affd640 10 client.4422.objecter ms_dispatch 0x7f7c7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:52.484 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.496+0000 7f7c6affd640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:52.484 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.496+0000 7f7c6affd640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:52.484 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.497+0000 7f7c6affd640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-29T11:24:52.484 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.501+0000 7f7c7a59c640 10 monclient: get_auth_request con 0x7f7c5c06e0e0 auth_method 0 2024-03-29T11:24:52.486 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.503+0000 7f7c6affd640 10 client.4422.objecter ms_handle_connect 0x7f7c5c06e0e0 2024-03-29T11:24:52.486 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.503+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c700502d0 con 0x7f7c7c1457f0 2024-03-29T11:24:52.486 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.503+0000 7f7c6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:52.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.503+0000 7f7c6affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:52.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.832+0000 7f7c8080f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:52.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.832+0000 7f7c8080f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:52.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.832+0000 7f7c8080f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:24:52.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.832+0000 7f7c8080f640 1 -- 172.21.15.59:0/2552889694 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c7c0b3680 con 0x7f7c7c1457f0 2024-03-29T11:24:52.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.834+0000 7f7c6affd640 1 -- 172.21.15.59:0/2552889694 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f7c700523d0 con 0x7f7c7c1457f0 2024-03-29T11:24:52.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.834+0000 7f7c6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:52.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.834+0000 7f7c6affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:52.817 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:24:52.817 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:52.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.838+0000 7f7c8080f640 10 librados: watch_flush enter 2024-03-29T11:24:52.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.839+0000 7f7c8080f640 10 librados: watch_flush exit 2024-03-29T11:24:52.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.839+0000 7f7c8080f640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-29T11:24:52.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.839+0000 7f7c8080f640 10 client.4422.objecter successfully canceled tick 2024-03-29T11:24:52.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.840+0000 7f7c8080f640 1 -- 172.21.15.59:0/2552889694 >> v1:172.21.15.172:6816/2420742773 conn(0x7f7c5c06e0e0 legacy=0x7f7c5c0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.840+0000 7f7c8080f640 10 monclient: shutdown 2024-03-29T11:24:52.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.840+0000 7f7c8080f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:52.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.840+0000 7f7c8080f640 1 -- 172.21.15.59:0/2552889694 >> v1:172.21.15.59:6789/0 conn(0x7f7c7c1457f0 legacy=0x7f7c7c1e2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.841+0000 7f7c8080f640 1 -- 172.21.15.59:0/2552889694 shutdown_connections 2024-03-29T11:24:52.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.841+0000 7f7c8080f640 1 -- 172.21.15.59:0/2552889694 >> 172.21.15.59:0/2552889694 conn(0x7f7c7c066a00 msgr2=0x7f7c7c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:52.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.843+0000 7f7c8080f640 1 -- 172.21.15.59:0/2552889694 shutdown_connections 2024-03-29T11:24:52.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.844+0000 7f7c8080f640 1 -- 172.21.15.59:0/2552889694 wait complete. 2024-03-29T11:24:52.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:52.845+0000 7f7c8080f640 1 librados: shutdown 2024-03-29T11:24:52.844 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:59.651 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:59.799 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.816+0000 7f5672231640 10 monclient: get_monmap_and_config 2024-03-29T11:24:59.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.816+0000 7f5672231640 10 monclient: build_initial_monmap 2024-03-29T11:24:59.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.816+0000 7f5672231640 10 monclient: monmap: 2024-03-29T11:24:59.800 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:59.800 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:59.800 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:24:59.817659+0000 2024-03-29T11:24:59.800 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:24:59.817659+0000 2024-03-29T11:24:59.800 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:59.800 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:59.800 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:59.800 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:24:59.801 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:24:59.801 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:59.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.817+0000 7f5672231640 1 Processor -- start 2024-03-29T11:24:59.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.818+0000 7f5672231640 1 -- start start 2024-03-29T11:24:59.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.818+0000 7f5672231640 10 monclient: init 2024-03-29T11:24:59.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f5672231640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:59.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f5672231640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:59.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f5672231640 10 monclient(hunting): picked mon.noname-b con 0x7f5664038c60 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:59.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f5672231640 10 monclient(hunting): picked mon.noname-a con 0x7f56640324b0 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:59.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f5672231640 10 monclient(hunting): picked mon.noname-c con 0x7f56640d11e0 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:59.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f5672231640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56640caf80 con 0x7f56640324b0 2024-03-29T11:24:59.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f5672231640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5664039850 con 0x7f56640d11e0 2024-03-29T11:24:59.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f5672231640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56640c78d0 con 0x7f5664038c60 2024-03-29T11:24:59.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f5672231640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:59.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f5672231640 10 monclient(hunting): authenticate will time out at 1149.932958s 2024-03-29T11:24:59.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f5671a30640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f56640d11e0 0x7f56640d35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:34836/0 (socket says 172.21.15.59:34836) 2024-03-29T11:24:59.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f567122f640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f5664038c60 0x7f56640c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:44268/0 (socket says 172.21.15.59:44268) 2024-03-29T11:24:59.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.819+0000 7f567122f640 1 -- 172.21.15.59:0/510005017 learned_addr learned my addr 172.21.15.59:0/510005017 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:59.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060721606 0 0) 0x7f5664039850 con 0x7f56640d11e0 2024-03-29T11:24:59.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:24:59.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5658003520 con 0x7f56640d11e0 2024-03-29T11:24:59.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490109637 0 0) 0x7f56640c78d0 con 0x7f5664038c60 2024-03-29T11:24:59.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:24:59.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5658003840 con 0x7f5664038c60 2024-03-29T11:24:59.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082479900 0 0) 0x7f56640caf80 con 0x7f56640324b0 2024-03-29T11:24:59.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:24:59.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.820+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56640c78d0 con 0x7f56640324b0 2024-03-29T11:24:59.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2302427905 0 0) 0x7f5658003520 con 0x7f56640d11e0 2024-03-29T11:24:59.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 >> v1:172.21.15.59:6790/0 conn(0x7f5664038c60 legacy=0x7f56640c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 >> v1:172.21.15.59:6789/0 conn(0x7f56640324b0 legacy=0x7f56640cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 1 monclient: found mon.noname-c 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56640c7a60 con 0x7f56640d11e0 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 10 monclient: _finish_auth 0 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f5672231640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f5672231640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5660001670 con 0x7f56640d11e0 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 10 monclient: dump: 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f5672231640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 1 -- 172.21.15.59:0/510005017 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5660001850 con 0x7f56640d11e0 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f565ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:59.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f5672231640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.821+0000 7f5672231640 4 set_mon_vals no callback set 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.822+0000 7f5672231640 10 monclient: shutdown 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.822+0000 7f5672231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.822+0000 7f5672231640 1 -- 172.21.15.59:0/510005017 >> v1:172.21.15.172:6789/0 conn(0x7f56640d11e0 legacy=0x7f56640d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.823+0000 7f5672231640 1 -- 172.21.15.59:0/510005017 shutdown_connections 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.823+0000 7f5672231640 1 -- 172.21.15.59:0/510005017 >> 172.21.15.59:0/510005017 conn(0x7f5664005c20 msgr2=0x7f5664006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.823+0000 7f5672231640 1 -- 172.21.15.59:0/510005017 shutdown_connections 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.823+0000 7f5672231640 1 -- 172.21.15.59:0/510005017 wait complete. 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.824+0000 7f5672231640 10 monclient: build_initial_monmap 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.824+0000 7f5672231640 10 monclient: monmap: 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:24:59.820 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:59.821 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:24:59.821 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:24:59.821 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:59.821 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:59.821 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:24:59.821 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:24:59.821 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:24:59.821 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:59.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.825+0000 7f5672231640 1 librados: starting msgr at 2024-03-29T11:24:59.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.825+0000 7f5672231640 1 librados: starting objecter 2024-03-29T11:24:59.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.825+0000 7f5672231640 1 Processor -- start 2024-03-29T11:24:59.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.825+0000 7f5672231640 1 -- start start 2024-03-29T11:24:59.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.825+0000 7f5672231640 1 librados: setting wanted keys 2024-03-29T11:24:59.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.825+0000 7f5672231640 1 librados: calling monclient init 2024-03-29T11:24:59.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.825+0000 7f5672231640 10 monclient: init 2024-03-29T11:24:59.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5672231640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:59.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5672231640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:59.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5672231640 10 monclient(hunting): picked mon.noname-a con 0x7f56640324b0 addr v1:172.21.15.59:6789/0 2024-03-29T11:24:59.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5672231640 10 monclient(hunting): picked mon.noname-b con 0x7f5664038c60 addr v1:172.21.15.172:6789/0 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5672231640 10 monclient(hunting): picked mon.noname-c con 0x7f56640d11e0 addr v1:172.21.15.59:6790/0 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5672231640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566402f6f0 con 0x7f56640324b0 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5672231640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566402f9d0 con 0x7f5664038c60 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5672231640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56640301d0 con 0x7f56640d11e0 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5672231640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5672231640 10 monclient(hunting): authenticate will time out at 1149.940198s 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5670a2e640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5664038c60 0x7f566402dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:36658/0 (socket says 172.21.15.59:36658) 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f5670a2e640 1 -- 172.21.15.59:0/1412478312 learned_addr learned my addr 172.21.15.59:0/1412478312 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f565dffb640 10 client.?.objecter ms_handle_connect 0x7f5664038c60 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.826+0000 7f565dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502280227 0 0) 0x7f566402f9d0 con 0x7f5664038c60 2024-03-29T11:24:59.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5650004eb0 con 0x7f5664038c60 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 client.?.objecter ms_handle_connect 0x7f56640d11e0 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 client.?.objecter ms_handle_connect 0x7f56640324b0 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721478115 0 0) 0x7f566402f6f0 con 0x7f56640324b0 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:24:59.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f566402f9d0 con 0x7f56640324b0 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2633328724 0 0) 0x7f5650004eb0 con 0x7f5664038c60 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f566402f6f0 con 0x7f5664038c60 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5668001a10 con 0x7f5664038c60 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): dump: 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:59.825 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.827+0000 7f565dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894844727 0 0) 0x7f566402f9d0 con 0x7f56640324b0 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5650004eb0 con 0x7f56640324b0 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4167648142 0 0) 0x7f566402f6f0 con 0x7f5664038c60 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 >> v1:172.21.15.59:6790/0 conn(0x7f56640d11e0 legacy=0x7f5664163370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 >> v1:172.21.15.59:6789/0 conn(0x7f56640324b0 legacy=0x7f566402d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 1 monclient: found mon.b 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566402fdf0 con 0x7f5664038c60 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:59.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:59.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:59.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f5672231640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:24:59.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f5672231640 10 monclient: _renew_subs 2024-03-29T11:24:59.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f5672231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:24:59.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f5672231640 1 -- 172.21.15.59:0/1412478312 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56640ba980 con 0x7f5664038c60 2024-03-29T11:24:59.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f5672231640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:59.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f5672231640 10 monclient: _renew_subs 2024-03-29T11:24:59.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f5672231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:24:59.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f5672231640 1 -- 172.21.15.59:0/1412478312 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5664036140 con 0x7f5664038c60 2024-03-29T11:24:59.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f5672231640 1 librados: init done 2024-03-29T11:24:59.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.828+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5668001d10 con 0x7f5664038c60 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.829+0000 7f565dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.829+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5668003a60 con 0x7f5664038c60 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.829+0000 7f565dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.829+0000 7f565dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.829+0000 7f565dffb640 10 monclient: dump: 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:24:59.828 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.829+0000 7f565f7fe640 4 set_mon_vals no callback set 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.830+0000 7f5672231640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.830+0000 7f5672231640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.830+0000 7f5672231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.830+0000 7f5672231640 1 -- 172.21.15.59:0/1412478312 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564c005080 con 0x7f5664038c60 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.831+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f5668003bc0 con 0x7f5664038c60 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.831+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f5668084b70 con 0x7f5664038c60 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.831+0000 7f565dffb640 10 client.4417.objecter ms_dispatch 0x7f566c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.831+0000 7f565dffb640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.831+0000 7f565dffb640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.831+0000 7f565dffb640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-29T11:24:59.829 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.831+0000 7f567122f640 10 monclient: get_auth_request con 0x7f565006dca0 auth_method 0 2024-03-29T11:24:59.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.832+0000 7f565dffb640 10 client.4417.objecter ms_handle_connect 0x7f565006dca0 2024-03-29T11:24:59.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.839+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5668050450 con 0x7f5664038c60 2024-03-29T11:24:59.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.839+0000 7f565dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:59.831 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:24:59.839+0000 7f565dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:00.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.245+0000 7f5672231640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.245+0000 7f5672231640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.245+0000 7f5672231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:00.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.245+0000 7f5672231640 1 -- 172.21.15.59:0/1412478312 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f564c005330 con 0x7f5664038c60 2024-03-29T11:25:00.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.246+0000 7f565dffb640 1 -- 172.21.15.59:0/1412478312 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f5668052380 con 0x7f5664038c60 2024-03-29T11:25:00.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.246+0000 7f565dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.246+0000 7f565dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:00.230 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:25:00.230 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:00.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.250+0000 7f5672231640 10 librados: watch_flush enter 2024-03-29T11:25:00.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.250+0000 7f5672231640 10 librados: watch_flush exit 2024-03-29T11:25:00.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.250+0000 7f5672231640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-29T11:25:00.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.250+0000 7f5672231640 10 client.4417.objecter successfully canceled tick 2024-03-29T11:25:00.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.250+0000 7f5672231640 1 -- 172.21.15.59:0/1412478312 >> v1:172.21.15.172:6816/2420742773 conn(0x7f565006dca0 legacy=0x7f5650070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:00.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.250+0000 7f5672231640 10 monclient: shutdown 2024-03-29T11:25:00.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.250+0000 7f5672231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:00.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.250+0000 7f5672231640 1 -- 172.21.15.59:0/1412478312 >> v1:172.21.15.172:6789/0 conn(0x7f5664038c60 legacy=0x7f566402dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:00.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.251+0000 7f5672231640 1 -- 172.21.15.59:0/1412478312 shutdown_connections 2024-03-29T11:25:00.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.251+0000 7f5672231640 1 -- 172.21.15.59:0/1412478312 >> 172.21.15.59:0/1412478312 conn(0x7f5664005c20 msgr2=0x7f5664006000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:00.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.251+0000 7f5672231640 1 -- 172.21.15.59:0/1412478312 shutdown_connections 2024-03-29T11:25:00.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.251+0000 7f5672231640 1 -- 172.21.15.59:0/1412478312 wait complete. 2024-03-29T11:25:00.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:00.251+0000 7f5672231640 1 librados: shutdown 2024-03-29T11:25:00.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:07.056 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:07.207 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.223+0000 7f548c4a8640 10 monclient: get_monmap_and_config 2024-03-29T11:25:07.207 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.223+0000 7f548c4a8640 10 monclient: build_initial_monmap 2024-03-29T11:25:07.207 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.223+0000 7f548c4a8640 10 monclient: monmap: 2024-03-29T11:25:07.208 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:07.208 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:07.208 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:25:07.224679+0000 2024-03-29T11:25:07.208 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:25:07.224679+0000 2024-03-29T11:25:07.208 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:07.208 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:07.208 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:07.208 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:25:07.208 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:25:07.208 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.224+0000 7f548c4a8640 1 Processor -- start 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.225+0000 7f548c4a8640 1 -- start start 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.225+0000 7f548c4a8640 10 monclient: init 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548c4a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548c4a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548c4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5484070120 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548c4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f54840b0420 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548c4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5484070500 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548c4a8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5484072090 con 0x7f54840b0420 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548c4a8640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5484072370 con 0x7f5484070500 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548c4a8640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5484072650 con 0x7f5484070120 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548c4a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548c4a8640 10 monclient(hunting): authenticate will time out at 1157.340392s 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548aa45640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5484070500 0x7f548414b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:48332/0 (socket says 172.21.15.59:48332) 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.226+0000 7f548aa45640 1 -- 172.21.15.59:0/736787265 learned_addr learned my addr 172.21.15.59:0/736787265 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.227+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855567888 0 0) 0x7f5484072370 con 0x7f5484070500 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.227+0000 7f5489242640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.227+0000 7f5489242640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.227+0000 7f5489242640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.227+0000 7f5489242640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.227+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5470003520 con 0x7f5484070500 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544861252 0 0) 0x7f5484072090 con 0x7f54840b0420 2024-03-29T11:25:07.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:25:07.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5484072370 con 0x7f54840b0420 2024-03-29T11:25:07.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384559799 0 0) 0x7f5484072650 con 0x7f5484070120 2024-03-29T11:25:07.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54700039d0 con 0x7f5484070120 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 217796573 0 0) 0x7f5470003520 con 0x7f5484070500 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 >> v1:172.21.15.59:6790/0 conn(0x7f5484070120 legacy=0x7f548413d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 >> v1:172.21.15.59:6789/0 conn(0x7f54840b0420 legacy=0x7f54841405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient: _finish_hunting 0 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 monclient: found mon.noname-c 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5484140090 con 0x7f5484070500 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient: _finish_auth 0 2024-03-29T11:25:07.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient: _check_auth_tickets 2024-03-29T11:25:07.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:07.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5480001690 con 0x7f5484070500 2024-03-29T11:25:07.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:07.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient: dump: 2024-03-29T11:25:07.219 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:07.219 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:07.219 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:07.219 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:07.219 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f548c4a8640 5 monclient: authenticate success, global_id 4420 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f548c4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5480001870 con 0x7f5484070500 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 1 -- 172.21.15.59:0/736787265 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5480001690 con 0x7f5484070500 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.228+0000 7f5489242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.229+0000 7f5489242640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:07.220 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.229+0000 7f5489242640 10 monclient: dump: 2024-03-29T11:25:07.221 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:07.222 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:07.222 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:07.222 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:07.222 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.222 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:07.222 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:07.222 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:07.222 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:07.222 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:07.222 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.229+0000 7f548c4a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:07.222 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.229+0000 7f548c4a8640 4 set_mon_vals no callback set 2024-03-29T11:25:07.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.229+0000 7f548c4a8640 10 monclient: shutdown 2024-03-29T11:25:07.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.229+0000 7f548c4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:07.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.229+0000 7f548c4a8640 1 -- 172.21.15.59:0/736787265 >> v1:172.21.15.172:6789/0 conn(0x7f5484070500 legacy=0x7f548414b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.230+0000 7f548c4a8640 1 -- 172.21.15.59:0/736787265 shutdown_connections 2024-03-29T11:25:07.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.230+0000 7f548c4a8640 1 -- 172.21.15.59:0/736787265 >> 172.21.15.59:0/736787265 conn(0x7f5484067500 msgr2=0x7f54840b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:07.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.230+0000 7f548c4a8640 1 -- 172.21.15.59:0/736787265 shutdown_connections 2024-03-29T11:25:07.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.230+0000 7f548c4a8640 1 -- 172.21.15.59:0/736787265 wait complete. 2024-03-29T11:25:07.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.231+0000 7f548c4a8640 10 monclient: build_initial_monmap 2024-03-29T11:25:07.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.231+0000 7f548c4a8640 10 monclient: monmap: 2024-03-29T11:25:07.223 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:07.223 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.231+0000 7f548c4a8640 1 librados: starting msgr at 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.231+0000 7f548c4a8640 1 librados: starting objecter 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.231+0000 7f548c4a8640 1 Processor -- start 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.232+0000 7f548c4a8640 1 -- start start 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.232+0000 7f548c4a8640 1 librados: setting wanted keys 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.232+0000 7f548c4a8640 1 librados: calling monclient init 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.232+0000 7f548c4a8640 10 monclient: init 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548c4a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548c4a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548c4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5484070120 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548c4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5484070500 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548c4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f54840b0420 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548c4a8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54840aa060 con 0x7f5484070500 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548a244640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5484070120 0x7f54841d7060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:48668/0 (socket says 172.21.15.59:48668) 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548c4a8640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54841d68f0 con 0x7f5484070120 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548a244640 1 -- 172.21.15.59:0/4158930963 learned_addr learned my addr 172.21.15.59:0/4158930963 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548c4a8640 1 -- 172.21.15.59:0/4158930963 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54841d7770 con 0x7f54840b0420 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548c4a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f548c4a8640 10 monclient(hunting): authenticate will time out at 1157.347372s 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f5489a43640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f5484070500 0x7f54841ca4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59584/0 (socket says 172.21.15.59:59584) 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f547affd640 10 client.?.objecter ms_handle_connect 0x7f5484070120 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.233+0000 7f547affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632729553 0 0) 0x7f54841d68f0 con 0x7f5484070120 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5468004eb0 con 0x7f5484070120 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 client.?.objecter ms_handle_connect 0x7f54840b0420 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 client.?.objecter ms_handle_connect 0x7f5484070500 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1578517501 0 0) 0x7f5468004eb0 con 0x7f5484070120 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54841d68f0 con 0x7f5484070120 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f547c001db0 con 0x7f5484070120 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 monclient(hunting): dump: 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.234+0000 7f547affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 93544141 0 0) 0x7f54841d68f0 con 0x7f5484070120 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 >> v1:172.21.15.59:6790/0 conn(0x7f54840b0420 legacy=0x7f5484138c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 >> v1:172.21.15.59:6789/0 conn(0x7f5484070500 legacy=0x7f54841ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f547affd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f547affd640 1 monclient: found mon.b 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f547affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f547affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54840aa2b0 con 0x7f5484070120 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f547affd640 10 monclient: _finish_auth 0 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f547affd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f547affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f548c4a8640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f548c4a8640 10 monclient: _renew_subs 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f548c4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f548c4a8640 1 -- 172.21.15.59:0/4158930963 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54841d6b40 con 0x7f5484070120 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f548c4a8640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f548c4a8640 10 monclient: _renew_subs 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f548c4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f548c4a8640 1 -- 172.21.15.59:0/4158930963 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54841cade0 con 0x7f5484070120 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.235+0000 7f548c4a8640 1 librados: init done 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.236+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f547c002030 con 0x7f5484070120 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.236+0000 7f547affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.236+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f547c003de0 con 0x7f5484070120 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.236+0000 7f547affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.236+0000 7f547affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.236+0000 7f547affd640 10 monclient: dump: 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.236+0000 7f547bfff640 4 set_mon_vals no callback set 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.237+0000 7f548c4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.237+0000 7f548c4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.237+0000 7f548c4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.237+0000 7f548c4a8640 1 -- 172.21.15.59:0/4158930963 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546c005080 con 0x7f5484070120 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.239+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f547c003f40 con 0x7f5484070120 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.239+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f547c084f10 con 0x7f5484070120 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.239+0000 7f547affd640 10 client.4423.objecter ms_dispatch 0x7f5484000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.239+0000 7f547affd640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.239+0000 7f547affd640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.239+0000 7f547affd640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.240+0000 7f5489a43640 10 monclient: get_auth_request con 0x7f546806dc20 auth_method 0 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.240+0000 7f547affd640 10 client.4423.objecter ms_handle_connect 0x7f546806dc20 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.246+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f547c0507f0 con 0x7f5484070120 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.246+0000 7f547affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.246+0000 7f547affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:07.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.657+0000 7f548c4a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:07.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.657+0000 7f548c4a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:07.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.657+0000 7f548c4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:07.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.657+0000 7f548c4a8640 1 -- 172.21.15.59:0/4158930963 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f546c005330 con 0x7f5484070120 2024-03-29T11:25:07.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.658+0000 7f547affd640 1 -- 172.21.15.59:0/4158930963 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f547c052720 con 0x7f5484070120 2024-03-29T11:25:07.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.659+0000 7f547affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:07.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.659+0000 7f547affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:07.642 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:25:07.642 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:07.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.662+0000 7f548c4a8640 10 librados: watch_flush enter 2024-03-29T11:25:07.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.662+0000 7f548c4a8640 10 librados: watch_flush exit 2024-03-29T11:25:07.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.662+0000 7f548c4a8640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:25:07.645 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.662+0000 7f548c4a8640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:25:07.646 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.662+0000 7f548c4a8640 1 -- 172.21.15.59:0/4158930963 >> v1:172.21.15.172:6816/2420742773 conn(0x7f546806dc20 legacy=0x7f54680700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.647 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.663+0000 7f548c4a8640 10 monclient: shutdown 2024-03-29T11:25:07.647 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.663+0000 7f548c4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:07.647 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.663+0000 7f548c4a8640 1 -- 172.21.15.59:0/4158930963 >> v1:172.21.15.172:6789/0 conn(0x7f5484070120 legacy=0x7f54841d7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.647 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.663+0000 7f548c4a8640 1 -- 172.21.15.59:0/4158930963 shutdown_connections 2024-03-29T11:25:07.647 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.663+0000 7f548c4a8640 1 -- 172.21.15.59:0/4158930963 >> 172.21.15.59:0/4158930963 conn(0x7f5484067500 msgr2=0x7f548406c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:07.647 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.663+0000 7f548c4a8640 1 -- 172.21.15.59:0/4158930963 shutdown_connections 2024-03-29T11:25:07.647 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.663+0000 7f548c4a8640 1 -- 172.21.15.59:0/4158930963 wait complete. 2024-03-29T11:25:07.647 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:07.663+0000 7f548c4a8640 1 librados: shutdown 2024-03-29T11:25:07.663 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:14.471 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:14.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.638+0000 7f8a1d397640 10 monclient: get_monmap_and_config 2024-03-29T11:25:14.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.638+0000 7f8a1d397640 10 monclient: build_initial_monmap 2024-03-29T11:25:14.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.638+0000 7f8a1d397640 10 monclient: monmap: 2024-03-29T11:25:14.623 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:14.623 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:14.623 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:25:14.639709+0000 2024-03-29T11:25:14.623 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:25:14.639709+0000 2024-03-29T11:25:14.623 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:14.624 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:14.624 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:14.624 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:25:14.624 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:25:14.624 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.640+0000 7f8a1d397640 1 Processor -- start 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.640+0000 7f8a1d397640 1 -- start start 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.640+0000 7f8a1d397640 10 monclient: init 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.641+0000 7f8a1d397640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.641+0000 7f8a1d397640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.641+0000 7f8a1d397640 10 monclient(hunting): picked mon.noname-b con 0x7f8a180b46b0 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.641+0000 7f8a1d397640 10 monclient(hunting): picked mon.noname-c con 0x7f8a180aeae0 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.641+0000 7f8a1d397640 10 monclient(hunting): picked mon.noname-a con 0x7f8a1813fa10 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.641+0000 7f8a1d397640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180b06a0 con 0x7f8a1813fa10 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.641+0000 7f8a1d397640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180b4230 con 0x7f8a180aeae0 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.641+0000 7f8a1d397640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180b0910 con 0x7f8a180b46b0 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.641+0000 7f8a1d397640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:14.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.641+0000 7f8a1d397640 10 monclient(hunting): authenticate will time out at 1164.755485s 2024-03-29T11:25:14.628 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.645+0000 7f8a16ffd640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f8a180b46b0 0x7f8a180b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:49709/0 (socket says 172.21.15.59:49709) 2024-03-29T11:25:14.629 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.645+0000 7f8a16ffd640 1 -- 172.21.15.59:0/4177118400 learned_addr learned my addr 172.21.15.59:0/4177118400 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:14.635 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 1 -- 172.21.15.59:0/4177118400 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763473782 0 0) 0x7f8a180b06a0 con 0x7f8a1813fa10 2024-03-29T11:25:14.635 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:14.635 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:14.635 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:14.635 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:25:14.635 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 1 -- 172.21.15.59:0/4177118400 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a00003520 con 0x7f8a1813fa10 2024-03-29T11:25:14.635 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 1 -- 172.21.15.59:0/4177118400 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679355550 0 0) 0x7f8a180b0910 con 0x7f8a180b46b0 2024-03-29T11:25:14.635 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:14.635 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:14.635 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 1 -- 172.21.15.59:0/4177118400 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a180b06a0 con 0x7f8a180b46b0 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 1 -- 172.21.15.59:0/4177118400 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 249836254 0 0) 0x7f8a00003520 con 0x7f8a1813fa10 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 1 -- 172.21.15.59:0/4177118400 >> v1:172.21.15.59:6790/0 conn(0x7f8a180b46b0 legacy=0x7f8a180b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 1 -- 172.21.15.59:0/4177118400 >> v1:172.21.15.172:6789/0 conn(0x7f8a180aeae0 legacy=0x7f8a1813f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 1 monclient: found mon.noname-a 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 1 -- 172.21.15.59:0/4177118400 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a180af6d0 con 0x7f8a1813fa10 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:14.636 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 1 -- 172.21.15.59:0/4177118400 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8a0c0015c0 con 0x7f8a1813fa10 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a15ffb640 10 monclient: dump: 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:14.637 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a1d397640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:25:14.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.651+0000 7f8a1d397640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:14.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.652+0000 7f8a15ffb640 1 -- 172.21.15.59:0/4177118400 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a0c001720 con 0x7f8a1813fa10 2024-03-29T11:25:14.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.652+0000 7f8a15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:14.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.652+0000 7f8a15ffb640 1 -- 172.21.15.59:0/4177118400 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8a0c001b30 con 0x7f8a1813fa10 2024-03-29T11:25:14.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.652+0000 7f8a15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.652+0000 7f8a15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:14.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.652+0000 7f8a15ffb640 10 monclient: dump: 2024-03-29T11:25:14.639 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:14.639 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:14.639 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:14.639 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:14.639 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.639 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:14.639 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:14.639 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.652+0000 7f8a1d397640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.652+0000 7f8a1d397640 4 set_mon_vals no callback set 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.652+0000 7f8a1d397640 10 monclient: shutdown 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.652+0000 7f8a1d397640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.652+0000 7f8a1d397640 1 -- 172.21.15.59:0/4177118400 >> v1:172.21.15.59:6789/0 conn(0x7f8a1813fa10 legacy=0x7f8a1814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.653+0000 7f8a1d397640 1 -- 172.21.15.59:0/4177118400 shutdown_connections 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.653+0000 7f8a1d397640 1 -- 172.21.15.59:0/4177118400 >> 172.21.15.59:0/4177118400 conn(0x7f8a18066a00 msgr2=0x7f8a18066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.654+0000 7f8a1d397640 1 -- 172.21.15.59:0/4177118400 shutdown_connections 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.654+0000 7f8a1d397640 1 -- 172.21.15.59:0/4177118400 wait complete. 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.655+0000 7f8a1d397640 10 monclient: build_initial_monmap 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.655+0000 7f8a1d397640 10 monclient: monmap: 2024-03-29T11:25:14.640 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.655+0000 7f8a1d397640 1 librados: starting msgr at 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.655+0000 7f8a1d397640 1 librados: starting objecter 2024-03-29T11:25:14.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.655+0000 7f8a1d397640 1 Processor -- start 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 1 -- start start 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 1 librados: setting wanted keys 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 1 librados: calling monclient init 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 10 monclient: init 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 10 monclient(hunting): picked mon.noname-b con 0x7f8a180aeae0 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 10 monclient(hunting): picked mon.noname-c con 0x7f8a180b46b0 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 10 monclient(hunting): picked mon.noname-a con 0x7f8a1813fa10 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a181459e0 con 0x7f8a1813fa10 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a181cb5b0 con 0x7f8a180aeae0 2024-03-29T11:25:14.642 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a18137650 con 0x7f8a180b46b0 2024-03-29T11:25:14.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:14.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.656+0000 7f8a1d397640 10 monclient(hunting): authenticate will time out at 1164.770592s 2024-03-29T11:25:14.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.660+0000 7f8a167fc640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f8a180b46b0 0x7f8a181cb9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:49819/0 (socket says 172.21.15.59:49819) 2024-03-29T11:25:14.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.660+0000 7f8a167fc640 1 -- 172.21.15.59:0/2554570163 learned_addr learned my addr 172.21.15.59:0/2554570163 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:14.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.660+0000 7f89ff7fe640 10 client.?.objecter ms_handle_connect 0x7f8a180b46b0 2024-03-29T11:25:14.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.660+0000 7f89ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:14.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.660+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990836094 0 0) 0x7f8a18137650 con 0x7f8a180b46b0 2024-03-29T11:25:14.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.660+0000 7f89ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:14.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.660+0000 7f89ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:14.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.660+0000 7f89ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:14.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.660+0000 7f89ff7fe640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:25:14.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.660+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a04004eb0 con 0x7f8a180b46b0 2024-03-29T11:25:14.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 10 client.?.objecter ms_handle_connect 0x7f8a1813fa10 2024-03-29T11:25:14.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:14.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2290104776 0 0) 0x7f8a04004eb0 con 0x7f8a180b46b0 2024-03-29T11:25:14.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a18137650 con 0x7f8a180b46b0 2024-03-29T11:25:14.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8a10001d80 con 0x7f8a180b46b0 2024-03-29T11:25:14.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:14.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 10 monclient(hunting): dump: 2024-03-29T11:25:14.649 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:14.649 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261978226 0 0) 0x7f8a181459e0 con 0x7f8a1813fa10 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:14.650 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.663+0000 7f89ff7fe640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a04004eb0 con 0x7f8a1813fa10 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2158239540 0 0) 0x7f8a18137650 con 0x7f8a180b46b0 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 >> v1:172.21.15.172:6789/0 conn(0x7f8a180aeae0 legacy=0x7f8a18141a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 >> v1:172.21.15.59:6789/0 conn(0x7f8a1813fa10 legacy=0x7f8a181e6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 1 monclient: found mon.c 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a181cf1a0 con 0x7f8a180b46b0 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f89ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f8a1d397640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f8a1d397640 10 monclient: _renew_subs 2024-03-29T11:25:14.651 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f8a1d397640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:14.652 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f8a1d397640 1 -- 172.21.15.59:0/2554570163 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a18142150 con 0x7f8a180b46b0 2024-03-29T11:25:14.652 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f8a1d397640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:14.652 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f8a1d397640 10 monclient: _renew_subs 2024-03-29T11:25:14.652 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f8a1d397640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:14.652 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f8a1d397640 1 -- 172.21.15.59:0/2554570163 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a18146050 con 0x7f8a180b46b0 2024-03-29T11:25:14.652 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.664+0000 7f8a1d397640 1 librados: init done 2024-03-29T11:25:14.652 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.665+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a10002040 con 0x7f8a180b46b0 2024-03-29T11:25:14.652 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.665+0000 7f89ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:14.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.665+0000 7f8a157fa640 4 set_mon_vals no callback set 2024-03-29T11:25:14.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.666+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8a10003de0 con 0x7f8a180b46b0 2024-03-29T11:25:14.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.666+0000 7f89ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.666+0000 7f89ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:14.653 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.666+0000 7f89ff7fe640 10 monclient: dump: 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.666+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f8a10004000 con 0x7f8a180b46b0 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.667+0000 7f8a1d397640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.667+0000 7f8a1d397640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:14.654 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.667+0000 7f8a1d397640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:14.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.667+0000 7f8a1d397640 1 -- 172.21.15.59:0/2554570163 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f8005080 con 0x7f8a180b46b0 2024-03-29T11:25:14.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.667+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f8a10084f10 con 0x7f8a180b46b0 2024-03-29T11:25:14.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.667+0000 7f89ff7fe640 10 client.4436.objecter ms_dispatch 0x7f8a18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:14.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.667+0000 7f89ff7fe640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:14.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.667+0000 7f89ff7fe640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:14.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.667+0000 7f89ff7fe640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T11:25:14.655 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.671+0000 7f8a16ffd640 10 monclient: get_auth_request con 0x7f8a0406dd20 auth_method 0 2024-03-29T11:25:14.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.675+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a10050870 con 0x7f8a180b46b0 2024-03-29T11:25:14.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.675+0000 7f89ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:14.659 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.675+0000 7f89ff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:14.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:14.676+0000 7f89ff7fe640 10 client.4436.objecter ms_handle_connect 0x7f8a0406dd20 2024-03-29T11:25:14.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.002+0000 7f8a1d397640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:14.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.002+0000 7f8a1d397640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:14.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.002+0000 7f8a1d397640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:14.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.002+0000 7f8a1d397640 1 -- 172.21.15.59:0/2554570163 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89f8005330 con 0x7f8a180b46b0 2024-03-29T11:25:14.992 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.006+0000 7f89ff7fe640 1 -- 172.21.15.59:0/2554570163 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f8a100527a0 con 0x7f8a180b46b0 2024-03-29T11:25:14.992 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.006+0000 7f89ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:14.992 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.006+0000 7f89ff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:14.993 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:25:14.993 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:14.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.013+0000 7f8a1d397640 10 librados: watch_flush enter 2024-03-29T11:25:14.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.013+0000 7f8a1d397640 10 librados: watch_flush exit 2024-03-29T11:25:14.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.013+0000 7f8a1d397640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T11:25:14.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.013+0000 7f8a1d397640 10 client.4436.objecter successfully canceled tick 2024-03-29T11:25:14.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.013+0000 7f8a1d397640 1 -- 172.21.15.59:0/2554570163 >> v1:172.21.15.172:6816/2420742773 conn(0x7f8a0406dd20 legacy=0x7f8a040701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.013+0000 7f8a1d397640 10 monclient: shutdown 2024-03-29T11:25:14.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.013+0000 7f8a1d397640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:14.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.013+0000 7f8a1d397640 1 -- 172.21.15.59:0/2554570163 >> v1:172.21.15.59:6790/0 conn(0x7f8a180b46b0 legacy=0x7f8a181cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.014+0000 7f8a1d397640 1 -- 172.21.15.59:0/2554570163 shutdown_connections 2024-03-29T11:25:14.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.014+0000 7f8a1d397640 1 -- 172.21.15.59:0/2554570163 >> 172.21.15.59:0/2554570163 conn(0x7f8a18066a00 msgr2=0x7f8a18138520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:14.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.015+0000 7f8a1d397640 1 -- 172.21.15.59:0/2554570163 shutdown_connections 2024-03-29T11:25:14.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.015+0000 7f8a1d397640 1 -- 172.21.15.59:0/2554570163 wait complete. 2024-03-29T11:25:14.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:15.015+0000 7f8a1d397640 1 librados: shutdown 2024-03-29T11:25:15.017 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:21.823 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:21.971 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.987+0000 7ff8ed48f640 10 monclient: get_monmap_and_config 2024-03-29T11:25:21.971 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.987+0000 7ff8ed48f640 10 monclient: build_initial_monmap 2024-03-29T11:25:21.971 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.987+0000 7ff8ed48f640 10 monclient: monmap: 2024-03-29T11:25:21.971 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:21.971 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:21.971 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:25:21.988084+0000 2024-03-29T11:25:21.971 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:25:21.988084+0000 2024-03-29T11:25:21.971 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:21.971 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:21.971 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:21.972 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:25:21.972 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:25:21.972 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:21.973 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.989+0000 7ff8ed48f640 1 Processor -- start 2024-03-29T11:25:21.974 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.989+0000 7ff8ed48f640 1 -- start start 2024-03-29T11:25:21.974 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.989+0000 7ff8ed48f640 10 monclient: init 2024-03-29T11:25:21.976 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.990+0000 7ff8ed48f640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:21.976 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.990+0000 7ff8ed48f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:21.976 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.990+0000 7ff8ed48f640 10 monclient(hunting): picked mon.noname-b con 0x7ff8e81457f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:21.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.990+0000 7ff8ed48f640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e813f650 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:21.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.990+0000 7ff8ed48f640 10 monclient(hunting): picked mon.noname-a con 0x7ff8e80a9d70 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:21.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.990+0000 7ff8ed48f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e80aaef0 con 0x7ff8e80a9d70 2024-03-29T11:25:21.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.990+0000 7ff8ed48f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e8145bd0 con 0x7ff8e813f650 2024-03-29T11:25:21.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.990+0000 7ff8ed48f640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e8145eb0 con 0x7ff8e81457f0 2024-03-29T11:25:21.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.990+0000 7ff8ed48f640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:21.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.991+0000 7ff8ed48f640 10 monclient(hunting): authenticate will time out at 1172.104690s 2024-03-29T11:25:21.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.991+0000 7ff8e67fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff8e813f650 0x7ff8e80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:54368/0 (socket says 172.21.15.59:54368) 2024-03-29T11:25:21.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.992+0000 7ff8e67fc640 1 -- 172.21.15.59:0/1463905377 learned_addr learned my addr 172.21.15.59:0/1463905377 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:21.977 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.992+0000 7ff8e5ffb640 1 -- 172.21.15.59:0/1463905377 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136985912 0 0) 0x7ff8e8145bd0 con 0x7ff8e813f650 2024-03-29T11:25:21.978 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.992+0000 7ff8e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:21.978 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.992+0000 7ff8e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:21.978 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.992+0000 7ff8e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:21.978 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.992+0000 7ff8e5ffb640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:25:21.978 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.992+0000 7ff8e5ffb640 1 -- 172.21.15.59:0/1463905377 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d0003520 con 0x7ff8e813f650 2024-03-29T11:25:21.978 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.992+0000 7ff8e5ffb640 1 -- 172.21.15.59:0/1463905377 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3098551164 0 0) 0x7ff8d0003520 con 0x7ff8e813f650 2024-03-29T11:25:21.978 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.992+0000 7ff8e5ffb640 1 -- 172.21.15.59:0/1463905377 >> v1:172.21.15.59:6790/0 conn(0x7ff8e81457f0 legacy=0x7ff8e813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:21.981 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.993+0000 7ff8e5ffb640 1 -- 172.21.15.59:0/1463905377 >> v1:172.21.15.59:6789/0 conn(0x7ff8e80a9d70 legacy=0x7ff8e814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:21.981 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:21.981 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:25:21.981 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:21.981 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:25:21.981 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 1 -- 172.21.15.59:0/1463905377 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e8146340 con 0x7ff8e813f650 2024-03-29T11:25:21.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:21.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:21.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:21.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 1 -- 172.21.15.59:0/1463905377 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff8d8001690 con 0x7ff8e813f650 2024-03-29T11:25:21.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8ed48f640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:25:21.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8ed48f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:21.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:21.982 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8e5ffb640 10 monclient: dump: 2024-03-29T11:25:21.982 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.995+0000 7ff8ed48f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.996+0000 7ff8e5ffb640 1 -- 172.21.15.59:0/1463905377 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8d8001920 con 0x7ff8e813f650 2024-03-29T11:25:21.983 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.996+0000 7ff8e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.996+0000 7ff8e5ffb640 1 -- 172.21.15.59:0/1463905377 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff8d8001cb0 con 0x7ff8e813f650 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.996+0000 7ff8e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.996+0000 7ff8e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.996+0000 7ff8e5ffb640 10 monclient: dump: 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:21.984 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:21.985 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:21.985 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.996+0000 7ff8ed48f640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:21.985 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.996+0000 7ff8ed48f640 4 set_mon_vals no callback set 2024-03-29T11:25:21.985 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.997+0000 7ff8ed48f640 10 monclient: shutdown 2024-03-29T11:25:21.985 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.997+0000 7ff8ed48f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:21.985 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.997+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1463905377 >> v1:172.21.15.172:6789/0 conn(0x7ff8e813f650 legacy=0x7ff8e80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:21.985 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.997+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1463905377 shutdown_connections 2024-03-29T11:25:21.985 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.997+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1463905377 >> 172.21.15.59:0/1463905377 conn(0x7ff8e8066a00 msgr2=0x7ff8e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:21.985 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.997+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1463905377 shutdown_connections 2024-03-29T11:25:21.985 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.997+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1463905377 wait complete. 2024-03-29T11:25:21.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.998+0000 7ff8ed48f640 10 monclient: build_initial_monmap 2024-03-29T11:25:21.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.998+0000 7ff8ed48f640 10 monclient: monmap: 2024-03-29T11:25:21.993 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:21.993 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:21.993 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.998+0000 7ff8ed48f640 1 librados: starting msgr at 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.998+0000 7ff8ed48f640 1 librados: starting objecter 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.999+0000 7ff8ed48f640 1 Processor -- start 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.999+0000 7ff8ed48f640 1 -- start start 2024-03-29T11:25:21.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.999+0000 7ff8ed48f640 1 librados: setting wanted keys 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.999+0000 7ff8ed48f640 1 librados: calling monclient init 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:21.999+0000 7ff8ed48f640 10 monclient: init 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.000+0000 7ff8ed48f640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.000+0000 7ff8ed48f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.000+0000 7ff8ed48f640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e80a9d70 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.000+0000 7ff8ed48f640 10 monclient(hunting): picked mon.noname-b con 0x7ff8e813f650 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.000+0000 7ff8ed48f640 10 monclient(hunting): picked mon.noname-a con 0x7ff8e81457f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.000+0000 7ff8ed48f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e8136d20 con 0x7ff8e81457f0 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.000+0000 7ff8ed48f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e806f550 con 0x7ff8e813f650 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.000+0000 7ff8ed48f640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e8136520 con 0x7ff8e80a9d70 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.000+0000 7ff8ed48f640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.000+0000 7ff8ed48f640 10 monclient(hunting): authenticate will time out at 1172.114053s 2024-03-29T11:25:21.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.001+0000 7ff8e67fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff8e813f650 0x7ff8e8070790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:54584/0 (socket says 172.21.15.59:54584) 2024-03-29T11:25:21.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.001+0000 7ff8e67fc640 1 -- 172.21.15.59:0/1539846182 learned_addr learned my addr 172.21.15.59:0/1539846182 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:21.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.002+0000 7ff8cf7fe640 10 client.?.objecter ms_handle_connect 0x7ff8e813f650 2024-03-29T11:25:21.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.002+0000 7ff8cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:21.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.002+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912644537 0 0) 0x7ff8e806f550 con 0x7ff8e813f650 2024-03-29T11:25:21.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.002+0000 7ff8cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:21.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.002+0000 7ff8cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:21.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.002+0000 7ff8cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:21.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.002+0000 7ff8cf7fe640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:25:21.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.002+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d4004eb0 con 0x7ff8e813f650 2024-03-29T11:25:21.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.003+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 745351781 0 0) 0x7ff8d4004eb0 con 0x7ff8e813f650 2024-03-29T11:25:21.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.003+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8e806f550 con 0x7ff8e813f650 2024-03-29T11:25:21.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.003+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff8d8001610 con 0x7ff8e813f650 2024-03-29T11:25:21.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.003+0000 7ff8cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.003+0000 7ff8cf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:21.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.003+0000 7ff8cf7fe640 10 monclient(hunting): dump: 2024-03-29T11:25:21.997 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:21.997 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:21.997 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:21.997 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:21.997 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.003+0000 7ff8cf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.004+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2885264779 0 0) 0x7ff8e806f550 con 0x7ff8e813f650 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.004+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 >> v1:172.21.15.59:6790/0 conn(0x7ff8e80a9d70 legacy=0x7ff8e806fd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.005+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 >> v1:172.21.15.59:6789/0 conn(0x7ff8e81457f0 legacy=0x7ff8e81e6040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.006+0000 7ff8cf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.006+0000 7ff8cf7fe640 1 monclient: found mon.b 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.006+0000 7ff8cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:21.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.006+0000 7ff8cf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.006+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e81366b0 con 0x7ff8e813f650 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.006+0000 7ff8cf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.006+0000 7ff8cf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.006+0000 7ff8cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8ed48f640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8ed48f640 10 monclient: _renew_subs 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8ed48f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1539846182 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e8070ea0 con 0x7ff8e813f650 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8ed48f640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8ed48f640 10 monclient: _renew_subs 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8ed48f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:21.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1539846182 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e8137320 con 0x7ff8e813f650 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8ed48f640 1 librados: init done 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8d8004bf0 con 0x7ff8e813f650 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff8d8004f40 con 0x7ff8e813f650 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8cf7fe640 10 monclient: dump: 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:22.000 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:22.001 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:22.001 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:22.001 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:22.001 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:22.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.007+0000 7ff8e4ff9640 4 set_mon_vals no callback set 2024-03-29T11:25:22.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.009+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7ff8d80050a0 con 0x7ff8e813f650 2024-03-29T11:25:22.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.009+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7ff8d8084d20 con 0x7ff8e813f650 2024-03-29T11:25:22.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.009+0000 7ff8cf7fe640 10 client.4432.objecter ms_dispatch 0x7ff8e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:22.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.009+0000 7ff8cf7fe640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:22.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.009+0000 7ff8cf7fe640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:22.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.009+0000 7ff8cf7fe640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-29T11:25:22.002 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.010+0000 7ff8ed48f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:22.002 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.010+0000 7ff8ed48f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:22.002 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.010+0000 7ff8ed48f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:22.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.010+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1539846182 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c8005080 con 0x7ff8e813f650 2024-03-29T11:25:22.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.010+0000 7ff8e6ffd640 10 monclient: get_auth_request con 0x7ff8d406dbe0 auth_method 0 2024-03-29T11:25:22.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.010+0000 7ff8cf7fe640 10 client.4432.objecter ms_handle_connect 0x7ff8d406dbe0 2024-03-29T11:25:22.005 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.020+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8d8050600 con 0x7ff8e813f650 2024-03-29T11:25:22.005 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.020+0000 7ff8cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:22.005 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.020+0000 7ff8cf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:22.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.361+0000 7ff8ed48f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.361+0000 7ff8ed48f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.361+0000 7ff8ed48f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:22.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.361+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1539846182 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8c8005330 con 0x7ff8e813f650 2024-03-29T11:25:22.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.361+0000 7ff8cf7fe640 1 -- 172.21.15.59:0/1539846182 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7ff8d8052530 con 0x7ff8e813f650 2024-03-29T11:25:22.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.361+0000 7ff8cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.361+0000 7ff8cf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:22.347 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:25:22.347 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:22.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.368+0000 7ff8ed48f640 10 librados: watch_flush enter 2024-03-29T11:25:22.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.368+0000 7ff8ed48f640 10 librados: watch_flush exit 2024-03-29T11:25:22.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.368+0000 7ff8ed48f640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-29T11:25:22.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.368+0000 7ff8ed48f640 10 client.4432.objecter successfully canceled tick 2024-03-29T11:25:22.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.368+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1539846182 >> v1:172.21.15.172:6816/2420742773 conn(0x7ff8d406dbe0 legacy=0x7ff8d40700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.368+0000 7ff8ed48f640 10 monclient: shutdown 2024-03-29T11:25:22.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.368+0000 7ff8ed48f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:22.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.368+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1539846182 >> v1:172.21.15.172:6789/0 conn(0x7ff8e813f650 legacy=0x7ff8e8070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.368+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1539846182 shutdown_connections 2024-03-29T11:25:22.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.368+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1539846182 >> 172.21.15.59:0/1539846182 conn(0x7ff8e8066a00 msgr2=0x7ff8e80b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:22.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.369+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1539846182 shutdown_connections 2024-03-29T11:25:22.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.369+0000 7ff8ed48f640 1 -- 172.21.15.59:0/1539846182 wait complete. 2024-03-29T11:25:22.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:22.369+0000 7ff8ed48f640 1 librados: shutdown 2024-03-29T11:25:22.368 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:29.176 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.343+0000 7f27657cf640 10 monclient: get_monmap_and_config 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.343+0000 7f27657cf640 10 monclient: build_initial_monmap 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.343+0000 7f27657cf640 10 monclient: monmap: 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:25:29.343815+0000 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:25:29.343815+0000 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.345+0000 7f27657cf640 1 Processor -- start 2024-03-29T11:25:29.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.346+0000 7f27657cf640 1 -- start start 2024-03-29T11:25:29.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.346+0000 7f27657cf640 10 monclient: init 2024-03-29T11:25:29.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.346+0000 7f27657cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:29.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.347+0000 7f27657cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:29.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.347+0000 7f27657cf640 10 monclient(hunting): picked mon.noname-a con 0x7f275802a490 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:29.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.347+0000 7f27657cf640 10 monclient(hunting): picked mon.noname-c con 0x7f275802b810 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:29.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.347+0000 7f27657cf640 10 monclient(hunting): picked mon.noname-b con 0x7f2758032ce0 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:29.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.347+0000 7f27657cf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275802aa40 con 0x7f275802a490 2024-03-29T11:25:29.331 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.347+0000 7f27657cf640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275802ca30 con 0x7f275802b810 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.347+0000 7f27657cf640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275802bc70 con 0x7f2758032ce0 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.347+0000 7f27657cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.347+0000 7f27657cf640 10 monclient(hunting): authenticate will time out at 1179.460805s 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.347+0000 7f2764fce640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f2758032ce0 0x7f27580c71e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:49662/0 (socket says 172.21.15.59:49662) 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.347+0000 7f2764fce640 1 -- 172.21.15.59:0/2921275064 learned_addr learned my addr 172.21.15.59:0/2921275064 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:29.334 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.350+0000 7f275effd640 1 -- 172.21.15.59:0/2921275064 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997150928 0 0) 0x7f275802bc70 con 0x7f2758032ce0 2024-03-29T11:25:29.334 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.350+0000 7f275effd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.334 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.350+0000 7f275effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.334 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.350+0000 7f275effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.334 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.350+0000 7f275effd640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:25:29.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.350+0000 7f275effd640 1 -- 172.21.15.59:0/2921275064 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2750003520 con 0x7f2758032ce0 2024-03-29T11:25:29.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 1 -- 172.21.15.59:0/2921275064 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2204419520 0 0) 0x7f2750003520 con 0x7f2758032ce0 2024-03-29T11:25:29.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 1 -- 172.21.15.59:0/2921275064 >> v1:172.21.15.172:6789/0 conn(0x7f275802b810 legacy=0x7f27580325d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 1 -- 172.21.15.59:0/2921275064 >> v1:172.21.15.59:6789/0 conn(0x7f275802a490 legacy=0x7f27580345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:29.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:29.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 1 monclient: found mon.noname-b 2024-03-29T11:25:29.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:29.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:25:29.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 1 -- 172.21.15.59:0/2921275064 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275802cca0 con 0x7f2758032ce0 2024-03-29T11:25:29.335 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 10 monclient: _finish_auth 0 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 1 -- 172.21.15.59:0/2921275064 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2754001670 con 0x7f2758032ce0 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f275effd640 10 monclient: dump: 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:29.336 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:29.337 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:29.337 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:29.337 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f27657cf640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:25:29.337 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.351+0000 7f27657cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:29.337 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.352+0000 7f275effd640 1 -- 172.21.15.59:0/2921275064 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27540017d0 con 0x7f2758032ce0 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.352+0000 7f275effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.352+0000 7f27657cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.352+0000 7f27657cf640 4 set_mon_vals no callback set 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.352+0000 7f27657cf640 10 monclient: shutdown 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.352+0000 7f27657cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.352+0000 7f27657cf640 1 -- 172.21.15.59:0/2921275064 >> v1:172.21.15.59:6790/0 conn(0x7f2758032ce0 legacy=0x7f27580c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.353+0000 7f275effd640 1 -- 172.21.15.59:0/2921275064 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2754001be0 con 0x7f2758032ce0 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.353+0000 7f275effd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.353+0000 7f27657cf640 1 -- 172.21.15.59:0/2921275064 shutdown_connections 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.353+0000 7f27657cf640 1 -- 172.21.15.59:0/2921275064 >> 172.21.15.59:0/2921275064 conn(0x7f27580058f0 msgr2=0x7f2758005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.353+0000 7f27657cf640 1 -- 172.21.15.59:0/2921275064 shutdown_connections 2024-03-29T11:25:29.338 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.353+0000 7f27657cf640 1 -- 172.21.15.59:0/2921275064 wait complete. 2024-03-29T11:25:29.339 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.354+0000 7f27657cf640 10 monclient: build_initial_monmap 2024-03-29T11:25:29.339 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.354+0000 7f27657cf640 10 monclient: monmap: 2024-03-29T11:25:29.339 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:29.339 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:29.339 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:25:29.339 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:25:29.339 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:29.339 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:29.339 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:29.340 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:25:29.340 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:25:29.340 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:29.340 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.354+0000 7f27657cf640 1 librados: starting msgr at 2024-03-29T11:25:29.340 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.354+0000 7f27657cf640 1 librados: starting objecter 2024-03-29T11:25:29.340 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.354+0000 7f27657cf640 1 Processor -- start 2024-03-29T11:25:29.340 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 1 -- start start 2024-03-29T11:25:29.340 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 1 librados: setting wanted keys 2024-03-29T11:25:29.340 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 1 librados: calling monclient init 2024-03-29T11:25:29.340 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 10 monclient: init 2024-03-29T11:25:29.340 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 10 monclient(hunting): picked mon.noname-c con 0x7f275802a490 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 10 monclient(hunting): picked mon.noname-a con 0x7f275802b810 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 10 monclient(hunting): picked mon.noname-b con 0x7f2758032ce0 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2758165870 con 0x7f275802b810 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27580c76e0 con 0x7f2758032ce0 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27580c0100 con 0x7f275802a490 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.355+0000 7f27657cf640 10 monclient(hunting): authenticate will time out at 1179.469660s 2024-03-29T11:25:29.343 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.358+0000 7f275ffff640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f275802a490 0x7f27580c0800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:51446/0 (socket says 172.21.15.59:51446) 2024-03-29T11:25:29.343 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.358+0000 7f275ffff640 1 -- 172.21.15.59:0/902811533 learned_addr learned my addr 172.21.15.59:0/902811533 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:29.343 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 client.?.objecter ms_handle_connect 0x7f275802b810 2024-03-29T11:25:29.343 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:29.343 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 client.?.objecter ms_handle_connect 0x7f2758032ce0 2024-03-29T11:25:29.343 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:29.343 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 client.?.objecter ms_handle_connect 0x7f275802a490 2024-03-29T11:25:29.344 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:29.344 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339580950 0 0) 0x7f27580c76e0 con 0x7f2758032ce0 2024-03-29T11:25:29.344 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.344 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2744004eb0 con 0x7f2758032ce0 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387277191 0 0) 0x7f27580c0100 con 0x7f275802a490 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2744006230 con 0x7f275802a490 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889606561 0 0) 0x7f2758165870 con 0x7f275802b810 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2744006570 con 0x7f275802b810 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.359+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4120336941 0 0) 0x7f2744006230 con 0x7f275802a490 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27440069e0 con 0x7f275802a490 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f276004fdd0 con 0x7f275802a490 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient(hunting): dump: 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.346 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 712260873 0 0) 0x7f2744004eb0 con 0x7f2758032ce0 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2744006230 con 0x7f2758032ce0 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2754001ae0 con 0x7f2758032ce0 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient(hunting): dump: 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:29.347 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:29.348 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.348 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:29.348 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:29.348 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:29.348 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:29.348 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2975392732 0 0) 0x7f27440069e0 con 0x7f275802a490 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 >> v1:172.21.15.172:6789/0 conn(0x7f2758032ce0 legacy=0x7f2758165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 >> v1:172.21.15.59:6789/0 conn(0x7f275802b810 legacy=0x7f2758164a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 monclient: found mon.c 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27580c03e0 con 0x7f275802a490 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:29.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f27657cf640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f27657cf640 10 monclient: _renew_subs 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f27657cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f27657cf640 1 -- 172.21.15.59:0/902811533 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27580c0f10 con 0x7f275802a490 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f27657cf640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f27657cf640 10 monclient: _renew_subs 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f27657cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f27657cf640 1 -- 172.21.15.59:0/902811533 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27580be890 con 0x7f275802a490 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f27657cf640 1 librados: init done 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f276004f400 con 0x7f275802a490 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2760075710 con 0x7f275802a490 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:29.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.360+0000 7f275cff9640 10 monclient: dump: 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.362+0000 7f275dffb640 4 set_mon_vals no callback set 2024-03-29T11:25:29.354 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.362+0000 7f274a7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.362+0000 7f274a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.362+0000 7f274a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:29.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.362+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f2760067bf0 con 0x7f275802a490 2024-03-29T11:25:29.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.362+0000 7f274a7fc640 1 -- 172.21.15.59:0/902811533 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2738005080 con 0x7f275802a490 2024-03-29T11:25:29.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.363+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f2760075870 con 0x7f275802a490 2024-03-29T11:25:29.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.363+0000 7f275cff9640 10 client.4442.objecter ms_dispatch 0x7f2760000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:29.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.363+0000 7f275cff9640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:29.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.363+0000 7f275cff9640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:29.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.363+0000 7f275cff9640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-29T11:25:29.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.363+0000 7f275f7fe640 10 monclient: get_auth_request con 0x7f274406e670 auth_method 0 2024-03-29T11:25:29.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.364+0000 7f275cff9640 10 client.4442.objecter ms_handle_connect 0x7f274406e670 2024-03-29T11:25:29.356 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.371+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27600cb5c0 con 0x7f275802a490 2024-03-29T11:25:29.356 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.371+0000 7f275cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.356 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.371+0000 7f275cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:29.684 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.700+0000 7f274a7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:29.684 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.700+0000 7f274a7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:29.684 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.700+0000 7f274a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:29.684 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.700+0000 7f274a7fc640 1 -- 172.21.15.59:0/902811533 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2738005330 con 0x7f275802a490 2024-03-29T11:25:29.685 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.701+0000 7f275cff9640 1 -- 172.21.15.59:0/902811533 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f27600cb5c0 con 0x7f275802a490 2024-03-29T11:25:29.685 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.701+0000 7f275cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:29.685 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.701+0000 7f275cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:29.685 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:25:29.686 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.705+0000 7f274a7fc640 10 librados: watch_flush enter 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.705+0000 7f274a7fc640 10 librados: watch_flush exit 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.705+0000 7f274a7fc640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.705+0000 7f274a7fc640 10 client.4442.objecter successfully canceled tick 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.705+0000 7f274a7fc640 1 -- 172.21.15.59:0/902811533 >> v1:172.21.15.172:6816/2420742773 conn(0x7f274406e670 legacy=0x7f2744070b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.705+0000 7f274a7fc640 10 monclient: shutdown 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.706+0000 7f274a7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.706+0000 7f274a7fc640 1 -- 172.21.15.59:0/902811533 >> v1:172.21.15.59:6790/0 conn(0x7f275802a490 legacy=0x7f27580c0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.706+0000 7f274a7fc640 1 -- 172.21.15.59:0/902811533 shutdown_connections 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.706+0000 7f274a7fc640 1 -- 172.21.15.59:0/902811533 >> 172.21.15.59:0/902811533 conn(0x7f27580058f0 msgr2=0x7f27580043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.706+0000 7f274a7fc640 1 -- 172.21.15.59:0/902811533 shutdown_connections 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.706+0000 7f274a7fc640 1 -- 172.21.15.59:0/902811533 wait complete. 2024-03-29T11:25:29.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:29.706+0000 7f274a7fc640 1 librados: shutdown 2024-03-29T11:25:29.708 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:36.515 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:36.661 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.675+0000 7f5389b6d640 10 monclient: get_monmap_and_config 2024-03-29T11:25:36.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.675+0000 7f5389b6d640 10 monclient: build_initial_monmap 2024-03-29T11:25:36.662 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.675+0000 7f5389b6d640 10 monclient: monmap: 2024-03-29T11:25:36.662 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:36.662 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:36.662 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:25:36.676468+0000 2024-03-29T11:25:36.662 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:25:36.676468+0000 2024-03-29T11:25:36.662 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:36.662 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:36.662 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:36.662 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.678+0000 7f5389b6d640 1 Processor -- start 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.678+0000 7f5389b6d640 1 -- start start 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.678+0000 7f5389b6d640 10 monclient: init 2024-03-29T11:25:36.665 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.679+0000 7f5389b6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:36.665 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.679+0000 7f5389b6d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.679+0000 7f5389b6d640 10 monclient(hunting): picked mon.noname-b con 0x7f53840b2800 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.679+0000 7f5389b6d640 10 monclient(hunting): picked mon.noname-c con 0x7f53840a9c00 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.679+0000 7f5389b6d640 10 monclient(hunting): picked mon.noname-a con 0x7f53840b38c0 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.679+0000 7f5389b6d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5384141eb0 con 0x7f53840b38c0 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.679+0000 7f5389b6d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53840b22d0 con 0x7f53840a9c00 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.679+0000 7f5389b6d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5384142d30 con 0x7f53840b2800 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.679+0000 7f5389b6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.679+0000 7f5389b6d640 10 monclient(hunting): authenticate will time out at 1186.792991s 2024-03-29T11:25:36.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.682+0000 7f5383fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f53840a9c00 0x7f538413bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:55488/0 (socket says 172.21.15.59:55488) 2024-03-29T11:25:36.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.682+0000 7f5383fff640 1 -- 172.21.15.59:0/1882546944 learned_addr learned my addr 172.21.15.59:0/1882546944 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:36.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.682+0000 7f53837fe640 1 -- 172.21.15.59:0/1882546944 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958051424 0 0) 0x7f5384141eb0 con 0x7f53840b38c0 2024-03-29T11:25:36.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:36.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:36.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:36.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:25:36.669 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 1 -- 172.21.15.59:0/1882546944 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5370003520 con 0x7f53840b38c0 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 1 -- 172.21.15.59:0/1882546944 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227837196 0 0) 0x7f5384142d30 con 0x7f53840b2800 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 1 -- 172.21.15.59:0/1882546944 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5384141eb0 con 0x7f53840b2800 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 1 -- 172.21.15.59:0/1882546944 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1272697441 0 0) 0x7f5370003520 con 0x7f53840b38c0 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 1 -- 172.21.15.59:0/1882546944 >> v1:172.21.15.59:6790/0 conn(0x7f53840b2800 legacy=0x7f5384142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 1 -- 172.21.15.59:0/1882546944 >> v1:172.21.15.172:6789/0 conn(0x7f53840a9c00 legacy=0x7f538413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 1 monclient: found mon.noname-a 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:36.670 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 1 -- 172.21.15.59:0/1882546944 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53840aaf90 con 0x7f53840b38c0 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 1 -- 172.21.15.59:0/1882546944 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f537c001690 con 0x7f53840b38c0 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f53837fe640 10 monclient: dump: 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:36.671 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f5389b6d640 5 monclient: authenticate success, global_id 4446 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.683+0000 7f5389b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.684+0000 7f53837fe640 1 -- 172.21.15.59:0/1882546944 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f537c0017f0 con 0x7f53840b38c0 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.684+0000 7f53837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.684+0000 7f53837fe640 1 -- 172.21.15.59:0/1882546944 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f537c001bc0 con 0x7f53840b38c0 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.684+0000 7f53837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.684+0000 7f53837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.684+0000 7f53837fe640 10 monclient: dump: 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:36.672 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.684+0000 7f5389b6d640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.684+0000 7f5389b6d640 4 set_mon_vals no callback set 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.685+0000 7f5389b6d640 10 monclient: shutdown 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.685+0000 7f5389b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.685+0000 7f5389b6d640 1 -- 172.21.15.59:0/1882546944 >> v1:172.21.15.59:6789/0 conn(0x7f53840b38c0 legacy=0x7f538414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.685+0000 7f5389b6d640 1 -- 172.21.15.59:0/1882546944 shutdown_connections 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.685+0000 7f5389b6d640 1 -- 172.21.15.59:0/1882546944 >> 172.21.15.59:0/1882546944 conn(0x7f5384066a00 msgr2=0x7f5384066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.687+0000 7f5389b6d640 1 -- 172.21.15.59:0/1882546944 shutdown_connections 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.687+0000 7f5389b6d640 1 -- 172.21.15.59:0/1882546944 wait complete. 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.688+0000 7f5389b6d640 10 monclient: build_initial_monmap 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.688+0000 7f5389b6d640 10 monclient: monmap: 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.689+0000 7f5389b6d640 1 librados: starting msgr at 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.689+0000 7f5389b6d640 1 librados: starting objecter 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.689+0000 7f5389b6d640 1 Processor -- start 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.689+0000 7f5389b6d640 1 -- start start 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.689+0000 7f5389b6d640 1 librados: setting wanted keys 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.689+0000 7f5389b6d640 1 librados: calling monclient init 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.689+0000 7f5389b6d640 10 monclient: init 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.690+0000 7f5389b6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.690+0000 7f5389b6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.690+0000 7f5389b6d640 10 monclient(hunting): picked mon.noname-b con 0x7f53840a9c00 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.693+0000 7f5388b6b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f53840a9c00 0x7f53841da7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:55938/0 (socket says 172.21.15.59:55938) 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.693+0000 7f5388b6b640 1 -- 172.21.15.59:0/4035460903 learned_addr learned my addr 172.21.15.59:0/4035460903 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.694+0000 7f53817fa640 10 client.?.objecter ms_handle_connect 0x7f53840a9c00 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.694+0000 7f53817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.694+0000 7f5389b6d640 10 monclient(hunting): picked mon.noname-c con 0x7f53840b2800 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.694+0000 7f5389b6d640 10 monclient(hunting): picked mon.noname-a con 0x7f53840b38c0 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.694+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53840afb00 con 0x7f53840b38c0 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.694+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53841daef0 con 0x7f53840a9c00 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.694+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53841db080 con 0x7f53840b2800 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.694+0000 7f5389b6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.694+0000 7f5389b6d640 10 monclient(hunting): authenticate will time out at 1186.808422s 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 10 client.?.objecter ms_handle_connect 0x7f53840b2800 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514391477 0 0) 0x7f53841db080 con 0x7f53840b2800 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5364004eb0 con 0x7f53840b2800 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119452192 0 0) 0x7f53841daef0 con 0x7f53840a9c00 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.697+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53841db080 con 0x7f53840a9c00 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 client.?.objecter ms_handle_connect 0x7f53840b38c0 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025973321 0 0) 0x7f53840afb00 con 0x7f53840b38c0 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53841daef0 con 0x7f53840b38c0 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1998006782 0 0) 0x7f53841db080 con 0x7f53840a9c00 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53840afb00 con 0x7f53840a9c00 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f536c001dd0 con 0x7f53840a9c00 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 monclient(hunting): dump: 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2124280001 0 0) 0x7f5364004eb0 con 0x7f53840b2800 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53841db080 con 0x7f53840b2800 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5374001510 con 0x7f53840b2800 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.698+0000 7f53817fa640 10 monclient(hunting): dump: 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2153113986 0 0) 0x7f53841db080 con 0x7f53840b2800 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 >> v1:172.21.15.172:6789/0 conn(0x7f53840a9c00 legacy=0x7f53841da7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 >> v1:172.21.15.59:6789/0 conn(0x7f53840b38c0 legacy=0x7f53840ae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f53817fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f53817fa640 1 monclient: found mon.c 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f53817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f53817fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53840b0270 con 0x7f53840b2800 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f53817fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f53817fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f53817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f5389b6d640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f5389b6d640 10 monclient: _renew_subs 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f5389b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:36.690 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53840b0460 con 0x7f53840b2800 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f5389b6d640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f5389b6d640 10 monclient: _renew_subs 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f5389b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f538413b560 con 0x7f53840b2800 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.699+0000 7f5389b6d640 1 librados: init done 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.700+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5374001a40 con 0x7f53840b2800 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.700+0000 7f53817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.700+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5374004b10 con 0x7f53840b2800 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.700+0000 7f53817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.700+0000 7f53817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.700+0000 7f53817fa640 10 monclient: dump: 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:36.693 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.700+0000 7f5382ffd640 4 set_mon_vals no callback set 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.701+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f537401aae0 con 0x7f53840b2800 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.702+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f5374084a20 con 0x7f53840b2800 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.702+0000 7f53817fa640 10 client.4448.objecter ms_dispatch 0x7f5384000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.702+0000 7f53817fa640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.702+0000 7f53817fa640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.702+0000 7f53817fa640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.702+0000 7f5388b6b640 10 monclient: get_auth_request con 0x7f53640767b0 auth_method 0 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.703+0000 7f53817fa640 10 client.4448.objecter ms_handle_connect 0x7f53640767b0 2024-03-29T11:25:36.694 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.703+0000 7f5389b6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:36.695 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.703+0000 7f5389b6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:36.695 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.703+0000 7f5389b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:36.695 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.703+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5368005080 con 0x7f53840b2800 2024-03-29T11:25:36.695 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.711+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5374050130 con 0x7f53840b2800 2024-03-29T11:25:36.695 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.711+0000 7f53817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:36.695 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:36.711+0000 7f53817fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:37.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.031+0000 7f5389b6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:37.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.032+0000 7f5389b6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:37.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.032+0000 7f5389b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:25:37.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.032+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5368005330 con 0x7f53840b2800 2024-03-29T11:25:37.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.034+0000 7f53817fa640 1 -- 172.21.15.59:0/4035460903 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f5374050130 con 0x7f53840b2800 2024-03-29T11:25:37.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.034+0000 7f53817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:37.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.034+0000 7f53817fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:37.019 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:25:37.019 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:37.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.038+0000 7f5389b6d640 10 librados: watch_flush enter 2024-03-29T11:25:37.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.038+0000 7f5389b6d640 10 librados: watch_flush exit 2024-03-29T11:25:37.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.038+0000 7f5389b6d640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T11:25:37.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.038+0000 7f5389b6d640 10 client.4448.objecter successfully canceled tick 2024-03-29T11:25:37.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.038+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 >> v1:172.21.15.172:6816/2420742773 conn(0x7f53640767b0 legacy=0x7f5364078c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.038+0000 7f5389b6d640 10 monclient: shutdown 2024-03-29T11:25:37.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.038+0000 7f5389b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:37.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.038+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 >> v1:172.21.15.59:6790/0 conn(0x7f53840b2800 legacy=0x7f538413ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.039+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 shutdown_connections 2024-03-29T11:25:37.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.039+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 >> 172.21.15.59:0/4035460903 conn(0x7f5384066a00 msgr2=0x7f53840b6940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:37.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.040+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 shutdown_connections 2024-03-29T11:25:37.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.040+0000 7f5389b6d640 1 -- 172.21.15.59:0/4035460903 wait complete. 2024-03-29T11:25:37.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:37.040+0000 7f5389b6d640 1 librados: shutdown 2024-03-29T11:25:37.040 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:43.848 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:43.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.011+0000 7f46e4a0c640 10 monclient: get_monmap_and_config 2024-03-29T11:25:43.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.011+0000 7f46e4a0c640 10 monclient: build_initial_monmap 2024-03-29T11:25:43.997 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.011+0000 7f46e4a0c640 10 monclient: monmap: 2024-03-29T11:25:43.997 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:43.997 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:43.997 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:25:44.013401+0000 2024-03-29T11:25:43.997 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:25:44.013401+0000 2024-03-29T11:25:43.997 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:43.997 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.012+0000 7f46e4a0c640 1 Processor -- start 2024-03-29T11:25:43.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.012+0000 7f46e4a0c640 1 -- start start 2024-03-29T11:25:43.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.012+0000 7f46e4a0c640 10 monclient: init 2024-03-29T11:25:43.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.013+0000 7f46e4a0c640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:43.999 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.013+0000 7f46e4a0c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:44.002 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.016+0000 7f46df7fe640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f46e00b2800 0x7f46e0142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:45017/0 (socket says 172.21.15.59:45017) 2024-03-29T11:25:44.002 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.016+0000 7f46df7fe640 1 -- 172.21.15.59:0/3755611109 learned_addr learned my addr 172.21.15.59:0/3755611109 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:44.002 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.016+0000 7f46e4a0c640 10 monclient(hunting): picked mon.noname-b con 0x7f46e00b2800 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:44.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.017+0000 7f46e4a0c640 10 monclient(hunting): picked mon.noname-c con 0x7f46e00a9c00 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:44.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.017+0000 7f46e4a0c640 10 monclient(hunting): picked mon.noname-a con 0x7f46e00b38c0 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:44.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.018+0000 7f46e4a0c640 1 -- 172.21.15.59:0/3755611109 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e0141eb0 con 0x7f46e00b38c0 2024-03-29T11:25:44.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.018+0000 7f46e4a0c640 1 -- 172.21.15.59:0/3755611109 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00b22d0 con 0x7f46e00a9c00 2024-03-29T11:25:44.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.018+0000 7f46e4a0c640 1 -- 172.21.15.59:0/3755611109 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e0142d30 con 0x7f46e00b2800 2024-03-29T11:25:44.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.018+0000 7f46e4a0c640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:44.003 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.018+0000 7f46e4a0c640 10 monclient(hunting): authenticate will time out at 1194.132727s 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 1 -- 172.21.15.59:0/3755611109 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944614217 0 0) 0x7f46e0141eb0 con 0x7f46e00b38c0 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 1 -- 172.21.15.59:0/3755611109 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c8003520 con 0x7f46e00b38c0 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 1 -- 172.21.15.59:0/3755611109 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985228829 0 0) 0x7f46e00b22d0 con 0x7f46e00a9c00 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 1 -- 172.21.15.59:0/3755611109 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e0141eb0 con 0x7f46e00a9c00 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 1 -- 172.21.15.59:0/3755611109 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3976014871 0 0) 0x7f46c8003520 con 0x7f46e00b38c0 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 1 -- 172.21.15.59:0/3755611109 >> v1:172.21.15.59:6790/0 conn(0x7f46e00b2800 legacy=0x7f46e0142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 1 -- 172.21.15.59:0/3755611109 >> v1:172.21.15.172:6789/0 conn(0x7f46e00a9c00 legacy=0x7f46e013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 1 monclient: found mon.noname-a 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.021+0000 7f46de7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46de7fc640 1 -- 172.21.15.59:0/3755611109 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e00aaf90 con 0x7f46e00b38c0 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46de7fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46de7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46de7fc640 1 -- 172.21.15.59:0/3755611109 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f46d8001a10 con 0x7f46e00b38c0 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46e4a0c640 5 monclient: authenticate success, global_id 4452 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46e4a0c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46de7fc640 10 monclient: dump: 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46e4a0c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46de7fc640 1 -- 172.21.15.59:0/3755611109 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46d8001c30 con 0x7f46e00b38c0 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46de7fc640 1 -- 172.21.15.59:0/3755611109 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f46d8001f40 con 0x7f46e00b38c0 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46e4a0c640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.022+0000 7f46e4a0c640 4 set_mon_vals no callback set 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.023+0000 7f46de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.023+0000 7f46de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.023+0000 7f46de7fc640 10 monclient: dump: 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:44.012 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:44.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.023+0000 7f46e4a0c640 10 monclient: shutdown 2024-03-29T11:25:44.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.023+0000 7f46e4a0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:44.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.023+0000 7f46e4a0c640 1 -- 172.21.15.59:0/3755611109 >> v1:172.21.15.59:6789/0 conn(0x7f46e00b38c0 legacy=0x7f46e014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.028+0000 7f46e4a0c640 1 -- 172.21.15.59:0/3755611109 shutdown_connections 2024-03-29T11:25:44.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.028+0000 7f46e4a0c640 1 -- 172.21.15.59:0/3755611109 >> 172.21.15.59:0/3755611109 conn(0x7f46e0066a00 msgr2=0x7f46e0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:44.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.031+0000 7f46e4a0c640 1 -- 172.21.15.59:0/3755611109 shutdown_connections 2024-03-29T11:25:44.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.031+0000 7f46e4a0c640 1 -- 172.21.15.59:0/3755611109 wait complete. 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.032+0000 7f46e4a0c640 10 monclient: build_initial_monmap 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.032+0000 7f46e4a0c640 10 monclient: monmap: 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.033+0000 7f46e4a0c640 1 librados: starting msgr at 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.033+0000 7f46e4a0c640 1 librados: starting objecter 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.033+0000 7f46e4a0c640 1 Processor -- start 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.034+0000 7f46e4a0c640 1 -- start start 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.034+0000 7f46e4a0c640 1 librados: setting wanted keys 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.034+0000 7f46e4a0c640 1 librados: calling monclient init 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.034+0000 7f46e4a0c640 10 monclient: init 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.035+0000 7f46e4a0c640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.035+0000 7f46e4a0c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.036+0000 7f46e4a0c640 10 monclient(hunting): picked mon.noname-b con 0x7f46e00a9c00 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.036+0000 7f46e4a0c640 10 monclient(hunting): picked mon.noname-a con 0x7f46e00b2800 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.036+0000 7f46e4a0c640 10 monclient(hunting): picked mon.noname-c con 0x7f46e00b38c0 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.037+0000 7f46e4a0c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00ad600 con 0x7f46e00b2800 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.037+0000 7f46e4a0c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e01db8a0 con 0x7f46e00a9c00 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.037+0000 7f46e4a0c640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e013f3f0 con 0x7f46e00b38c0 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.037+0000 7f46e4a0c640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.037+0000 7f46e4a0c640 10 monclient(hunting): authenticate will time out at 1194.151778s 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46deffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f46e00b2800 0x7f46e0137ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55329/0 (socket says 172.21.15.59:55329) 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46deffd640 1 -- 172.21.15.59:0/3744207557 learned_addr learned my addr 172.21.15.59:0/3744207557 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46c7fff640 10 client.?.objecter ms_handle_connect 0x7f46e00b2800 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46c7fff640 10 client.?.objecter ms_handle_connect 0x7f46e00a9c00 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604568295 0 0) 0x7f46e00ad600 con 0x7f46e00b2800 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46c7fff640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.040+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46cc004eb0 con 0x7f46e00b2800 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.041+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613379438 0 0) 0x7f46e01db8a0 con 0x7f46e00a9c00 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.041+0000 7f46c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.041+0000 7f46c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.041+0000 7f46c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.041+0000 7f46c7fff640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.041+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e00ad600 con 0x7f46e00a9c00 2024-03-29T11:25:44.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.044+0000 7f46c7fff640 10 client.?.objecter ms_handle_connect 0x7f46e00b38c0 2024-03-29T11:25:44.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.044+0000 7f46c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:44.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.044+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 754989151 0 0) 0x7f46cc004eb0 con 0x7f46e00b2800 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.044+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46e01db8a0 con 0x7f46e00b2800 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.044+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1683559440 0 0) 0x7f46e00ad600 con 0x7f46e00a9c00 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.044+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46cc004eb0 con 0x7f46e00a9c00 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.044+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f46d40019f0 con 0x7f46e00b2800 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.044+0000 7f46c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.044+0000 7f46c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.044+0000 7f46c7fff640 10 monclient(hunting): dump: 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.034 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.045+0000 7f46c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.045+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210408547 0 0) 0x7f46e013f3f0 con 0x7f46e00b38c0 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.045+0000 7f46c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.045+0000 7f46c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.045+0000 7f46c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.045+0000 7f46c7fff640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.045+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e00ad600 con 0x7f46e00b38c0 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.045+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f46d0001820 con 0x7f46e00a9c00 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:44.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 10 monclient(hunting): dump: 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3266331904 0 0) 0x7f46e01db8a0 con 0x7f46e00b2800 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 >> v1:172.21.15.59:6790/0 conn(0x7f46e00b38c0 legacy=0x7f46e00ab590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 >> v1:172.21.15.172:6789/0 conn(0x7f46e00a9c00 legacy=0x7f46e01dad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.036 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 1 monclient: found mon.a 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e013f580 con 0x7f46e00b2800 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46d4001b50 con 0x7f46e00b2800 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.046+0000 7f46c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f46d4003a60 con 0x7f46e00b2800 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46c7fff640 10 monclient: dump: 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:44.037 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46e4a0c640 5 monclient: authenticate success, global_id 4455 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46e4a0c640 10 monclient: _renew_subs 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46e4a0c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46e4a0c640 1 -- 172.21.15.59:0/3744207557 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e01db430 con 0x7f46e00b2800 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46e4a0c640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:44.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46e4a0c640 10 monclient: _renew_subs 2024-03-29T11:25:44.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46e4a0c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:25:44.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46e4a0c640 1 -- 172.21.15.59:0/3744207557 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e01387d0 con 0x7f46e00b2800 2024-03-29T11:25:44.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.047+0000 7f46e4a0c640 1 librados: init done 2024-03-29T11:25:44.040 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.048+0000 7f46dd7fa640 4 set_mon_vals no callback set 2024-03-29T11:25:44.040 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.050+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f46d4003bc0 con 0x7f46e00b2800 2024-03-29T11:25:44.040 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.050+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f46d4084aa0 con 0x7f46e00b2800 2024-03-29T11:25:44.040 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.050+0000 7f46c5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:44.040 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.050+0000 7f46c5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:44.040 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.050+0000 7f46c5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:25:44.040 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.050+0000 7f46c5ffb640 1 -- 172.21.15.59:0/3744207557 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b8005080 con 0x7f46e00b2800 2024-03-29T11:25:44.041 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.051+0000 7f46c7fff640 10 client.4455.objecter ms_dispatch 0x7f46e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:44.041 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.051+0000 7f46c7fff640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:44.041 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.051+0000 7f46c7fff640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:44.041 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.051+0000 7f46c7fff640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-29T11:25:44.041 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.053+0000 7f46df7fe640 10 monclient: get_auth_request con 0x7f46cc06de40 auth_method 0 2024-03-29T11:25:44.041 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.056+0000 7f46c7fff640 10 client.4455.objecter ms_handle_connect 0x7f46cc06de40 2024-03-29T11:25:44.045 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.060+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46d40501b0 con 0x7f46e00b2800 2024-03-29T11:25:44.045 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.060+0000 7f46c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:44.045 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.060+0000 7f46c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:44.368 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.383+0000 7f46c5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.383+0000 7f46c5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.383+0000 7f46c5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:25:44.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.383+0000 7f46c5ffb640 1 -- 172.21.15.59:0/3744207557 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46b8005330 con 0x7f46e00b2800 2024-03-29T11:25:44.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.384+0000 7f46c7fff640 1 -- 172.21.15.59:0/3744207557 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f46d40522b0 con 0x7f46e00b2800 2024-03-29T11:25:44.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.384+0000 7f46c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.384+0000 7f46c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:44.371 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:25:44.371 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:44.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.389+0000 7f46c5ffb640 10 librados: watch_flush enter 2024-03-29T11:25:44.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.389+0000 7f46c5ffb640 10 librados: watch_flush exit 2024-03-29T11:25:44.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.389+0000 7f46c5ffb640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-29T11:25:44.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.389+0000 7f46c5ffb640 10 client.4455.objecter successfully canceled tick 2024-03-29T11:25:44.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.389+0000 7f46c5ffb640 1 -- 172.21.15.59:0/3744207557 >> v1:172.21.15.172:6816/2420742773 conn(0x7f46cc06de40 legacy=0x7f46cc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.389+0000 7f46c5ffb640 10 monclient: shutdown 2024-03-29T11:25:44.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.389+0000 7f46c5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:44.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.389+0000 7f46c5ffb640 1 -- 172.21.15.59:0/3744207557 >> v1:172.21.15.59:6789/0 conn(0x7f46e00b2800 legacy=0x7f46e0137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.389+0000 7f46c5ffb640 1 -- 172.21.15.59:0/3744207557 shutdown_connections 2024-03-29T11:25:44.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.389+0000 7f46c5ffb640 1 -- 172.21.15.59:0/3744207557 >> 172.21.15.59:0/3744207557 conn(0x7f46e0066a00 msgr2=0x7f46e01405b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:44.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.389+0000 7f46c5ffb640 1 -- 172.21.15.59:0/3744207557 shutdown_connections 2024-03-29T11:25:44.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.390+0000 7f46c5ffb640 1 -- 172.21.15.59:0/3744207557 wait complete. 2024-03-29T11:25:44.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:44.390+0000 7f46c5ffb640 1 librados: shutdown 2024-03-29T11:25:44.391 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:51.198 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:51.351 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.365+0000 7fb7c509c640 10 monclient: get_monmap_and_config 2024-03-29T11:25:51.351 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.365+0000 7fb7c509c640 10 monclient: build_initial_monmap 2024-03-29T11:25:51.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.365+0000 7fb7c509c640 10 monclient: monmap: 2024-03-29T11:25:51.352 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:51.352 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:51.352 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:25:51.366965+0000 2024-03-29T11:25:51.352 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:25:51.366965+0000 2024-03-29T11:25:51.352 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:51.352 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:51.352 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:51.352 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:25:51.352 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:25:51.352 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:51.353 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.368+0000 7fb7c509c640 1 Processor -- start 2024-03-29T11:25:51.357 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.368+0000 7fb7c509c640 1 -- start start 2024-03-29T11:25:51.357 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.368+0000 7fb7c509c640 10 monclient: init 2024-03-29T11:25:51.357 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.369+0000 7fb7c509c640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:51.357 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.369+0000 7fb7c509c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:51.358 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.369+0000 7fb7c509c640 10 monclient(hunting): picked mon.noname-a con 0x7fb7c00a9ac0 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:51.358 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.369+0000 7fb7c509c640 10 monclient(hunting): picked mon.noname-b con 0x7fb7c013ee90 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:51.358 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.369+0000 7fb7c509c640 10 monclient(hunting): picked mon.noname-c con 0x7fb7c00af900 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:51.358 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.369+0000 7fb7c509c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c00afd10 con 0x7fb7c00a9ac0 2024-03-29T11:25:51.358 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.369+0000 7fb7c509c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c00af6f0 con 0x7fb7c00af900 2024-03-29T11:25:51.358 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.369+0000 7fb7c509c640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c00aac80 con 0x7fb7c013ee90 2024-03-29T11:25:51.358 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.369+0000 7fb7c509c640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:51.358 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.369+0000 7fb7c509c640 10 monclient(hunting): authenticate will time out at 1201.484536s 2024-03-29T11:25:51.359 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.373+0000 7fb7be59c640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fb7c013ee90 0x7fb7c013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:44141/0 (socket says 172.21.15.59:44141) 2024-03-29T11:25:51.359 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.373+0000 7fb7be59c640 1 -- 172.21.15.59:0/1921596206 learned_addr learned my addr 172.21.15.59:0/1921596206 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:51.360 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.374+0000 7fb7bdd9b640 1 -- 172.21.15.59:0/1921596206 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076710941 0 0) 0x7fb7c00aac80 con 0x7fb7c013ee90 2024-03-29T11:25:51.360 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.360 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.360 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.360 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:25:51.360 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 1 -- 172.21.15.59:0/1921596206 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a8004630 con 0x7fb7c013ee90 2024-03-29T11:25:51.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 1 -- 172.21.15.59:0/1921596206 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992563670 0 0) 0x7fb7c00afd10 con 0x7fb7c00a9ac0 2024-03-29T11:25:51.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 1 -- 172.21.15.59:0/1921596206 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c00aac80 con 0x7fb7c00a9ac0 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 1 -- 172.21.15.59:0/1921596206 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2714430829 0 0) 0x7fb7a8004630 con 0x7fb7c013ee90 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 1 -- 172.21.15.59:0/1921596206 >> v1:172.21.15.172:6789/0 conn(0x7fb7c00af900 legacy=0x7fb7c0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 1 -- 172.21.15.59:0/1921596206 >> v1:172.21.15.59:6789/0 conn(0x7fb7c00a9ac0 legacy=0x7fb7c00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 1 monclient: found mon.noname-b 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 1 -- 172.21.15.59:0/1921596206 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c00aaf60 con 0x7fb7c013ee90 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient: _finish_auth 0 2024-03-29T11:25:51.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7c509c640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7c509c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 1 -- 172.21.15.59:0/1921596206 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb7b4004c30 con 0x7fb7c013ee90 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 10 monclient: dump: 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:51.363 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7c509c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.375+0000 7fb7bdd9b640 1 -- 172.21.15.59:0/1921596206 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7b4004d90 con 0x7fb7c013ee90 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.376+0000 7fb7bdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.376+0000 7fb7bdd9b640 1 -- 172.21.15.59:0/1921596206 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb7b40013f0 con 0x7fb7c013ee90 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.376+0000 7fb7bdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.376+0000 7fb7bdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.376+0000 7fb7bdd9b640 10 monclient: dump: 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.376+0000 7fb7c509c640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.376+0000 7fb7c509c640 4 set_mon_vals no callback set 2024-03-29T11:25:51.367 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.377+0000 7fb7c509c640 10 monclient: shutdown 2024-03-29T11:25:51.367 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.377+0000 7fb7c509c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:51.367 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.377+0000 7fb7c509c640 1 -- 172.21.15.59:0/1921596206 >> v1:172.21.15.59:6790/0 conn(0x7fb7c013ee90 legacy=0x7fb7c013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.367 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.377+0000 7fb7c509c640 1 -- 172.21.15.59:0/1921596206 shutdown_connections 2024-03-29T11:25:51.367 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.377+0000 7fb7c509c640 1 -- 172.21.15.59:0/1921596206 >> 172.21.15.59:0/1921596206 conn(0x7fb7c00b35e0 msgr2=0x7fb7c00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:51.367 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.378+0000 7fb7c509c640 1 -- 172.21.15.59:0/1921596206 shutdown_connections 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.379+0000 7fb7c509c640 1 -- 172.21.15.59:0/1921596206 wait complete. 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.380+0000 7fb7c509c640 10 monclient: build_initial_monmap 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.380+0000 7fb7c509c640 10 monclient: monmap: 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:25:51.368 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.380+0000 7fb7c509c640 1 librados: starting msgr at 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.380+0000 7fb7c509c640 1 librados: starting objecter 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.381+0000 7fb7c509c640 1 Processor -- start 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.381+0000 7fb7c509c640 1 -- start start 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.381+0000 7fb7c509c640 1 librados: setting wanted keys 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.381+0000 7fb7c509c640 1 librados: calling monclient init 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.381+0000 7fb7c509c640 10 monclient: init 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.382+0000 7fb7c509c640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.382+0000 7fb7c509c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.382+0000 7fb7c509c640 10 monclient(hunting): picked mon.noname-a con 0x7fb7c00af900 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:51.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.382+0000 7fb7c509c640 10 monclient(hunting): picked mon.noname-b con 0x7fb7c013ee90 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:51.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.382+0000 7fb7c509c640 10 monclient(hunting): picked mon.noname-c con 0x7fb7c01397f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:51.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.382+0000 7fb7c509c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c006f120 con 0x7fb7c00af900 2024-03-29T11:25:51.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.382+0000 7fb7c509c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c01d2e30 con 0x7fb7c013ee90 2024-03-29T11:25:51.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.382+0000 7fb7c509c640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c01d2fc0 con 0x7fb7c01397f0 2024-03-29T11:25:51.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.382+0000 7fb7c509c640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:51.370 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.382+0000 7fb7c509c640 10 monclient(hunting): authenticate will time out at 1201.497020s 2024-03-29T11:25:51.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.386+0000 7fb7bf59e640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fb7c01397f0 0x7fb7c0139c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:44239/0 (socket says 172.21.15.59:44239) 2024-03-29T11:25:51.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.386+0000 7fb7bf59e640 1 -- 172.21.15.59:0/3929361457 learned_addr learned my addr 172.21.15.59:0/3929361457 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:51.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.386+0000 7fb7a77fe640 10 client.?.objecter ms_handle_connect 0x7fb7c01397f0 2024-03-29T11:25:51.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.386+0000 7fb7a77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:51.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 10 client.?.objecter ms_handle_connect 0x7fb7c013ee90 2024-03-29T11:25:51.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:51.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210179540 0 0) 0x7fb7c01d2fc0 con 0x7fb7c01397f0 2024-03-29T11:25:51.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:25:51.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7ac004eb0 con 0x7fb7c01397f0 2024-03-29T11:25:51.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957764038 0 0) 0x7fb7c01d2e30 con 0x7fb7c013ee90 2024-03-29T11:25:51.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.375 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:25:51.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.388+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c01d2fc0 con 0x7fb7c013ee90 2024-03-29T11:25:51.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 104022129 0 0) 0x7fb7c01d2fc0 con 0x7fb7c013ee90 2024-03-29T11:25:51.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7ac0064f0 con 0x7fb7c013ee90 2024-03-29T11:25:51.376 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb7b40013b0 con 0x7fb7c013ee90 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 10 monclient(hunting): dump: 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 21271418 0 0) 0x7fb7ac004eb0 con 0x7fb7c01397f0 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7ac006bc0 con 0x7fb7c01397f0 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb7b8001c60 con 0x7fb7c01397f0 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 10 monclient(hunting): dump: 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:51.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3445991139 0 0) 0x7fb7ac0064f0 con 0x7fb7c013ee90 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 >> v1:172.21.15.59:6790/0 conn(0x7fb7c01397f0 legacy=0x7fb7c0139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.389+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 >> v1:172.21.15.59:6789/0 conn(0x7fb7c00af900 legacy=0x7fb7c0141fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7a77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7a77fe640 1 monclient: found mon.b 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7a77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c01d3150 con 0x7fb7c013ee90 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7a77fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7a77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7c509c640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7c509c640 10 monclient: _renew_subs 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7c509c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7c509c640 1 -- 172.21.15.59:0/3929361457 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c01d33a0 con 0x7fb7c013ee90 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7c509c640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7c509c640 10 monclient: _renew_subs 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7c509c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7c509c640 1 -- 172.21.15.59:0/3929361457 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c01d3880 con 0x7fb7c013ee90 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.390+0000 7fb7c509c640 1 librados: init done 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.391+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7b4001960 con 0x7fb7c013ee90 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.391+0000 7fb7a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.391+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb7b40013b0 con 0x7fb7c013ee90 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.391+0000 7fb7a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.391+0000 7fb7a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.391+0000 7fb7a77fe640 10 monclient: dump: 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.391+0000 7fb7bcd99640 4 set_mon_vals no callback set 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.394+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fb7b4001bd0 con 0x7fb7c013ee90 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.394+0000 7fb7c509c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.394+0000 7fb7c509c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.394+0000 7fb7c509c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.394+0000 7fb7c509c640 1 -- 172.21.15.59:0/3929361457 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a0005080 con 0x7fb7c013ee90 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.394+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fb7b40543e0 con 0x7fb7c013ee90 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.394+0000 7fb7a77fe640 10 client.4450.objecter ms_dispatch 0x7fb7c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.394+0000 7fb7a77fe640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.394+0000 7fb7a77fe640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.395+0000 7fb7a77fe640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.395+0000 7fb7bed9d640 10 monclient: get_auth_request con 0x7fb7ac06e770 auth_method 0 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.397+0000 7fb7a77fe640 10 client.4450.objecter ms_handle_connect 0x7fb7ac06e770 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.404+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7b4050680 con 0x7fb7c013ee90 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.405+0000 7fb7a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.405+0000 7fb7a77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:51.722 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.737+0000 7fb7c509c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:51.723 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.737+0000 7fb7c509c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:51.723 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.737+0000 7fb7c509c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:25:51.723 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.737+0000 7fb7c509c640 1 -- 172.21.15.59:0/3929361457 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7a0005330 con 0x7fb7c013ee90 2024-03-29T11:25:51.724 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.737+0000 7fb7a77fe640 1 -- 172.21.15.59:0/3929361457 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fb7b4050820 con 0x7fb7c013ee90 2024-03-29T11:25:51.724 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.738+0000 7fb7a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:51.724 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.738+0000 7fb7a77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:51.724 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:25:51.724 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:51.727 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.741+0000 7fb7c509c640 10 librados: watch_flush enter 2024-03-29T11:25:51.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.741+0000 7fb7c509c640 10 librados: watch_flush exit 2024-03-29T11:25:51.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.741+0000 7fb7c509c640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-29T11:25:51.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.741+0000 7fb7c509c640 10 client.4450.objecter successfully canceled tick 2024-03-29T11:25:51.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.741+0000 7fb7c509c640 1 -- 172.21.15.59:0/3929361457 >> v1:172.21.15.172:6816/2420742773 conn(0x7fb7ac06e770 legacy=0x7fb7ac070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.741+0000 7fb7c509c640 10 monclient: shutdown 2024-03-29T11:25:51.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.741+0000 7fb7c509c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:51.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.741+0000 7fb7c509c640 1 -- 172.21.15.59:0/3929361457 >> v1:172.21.15.172:6789/0 conn(0x7fb7c013ee90 legacy=0x7fb7c006ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.742+0000 7fb7c509c640 1 -- 172.21.15.59:0/3929361457 shutdown_connections 2024-03-29T11:25:51.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.742+0000 7fb7c509c640 1 -- 172.21.15.59:0/3929361457 >> 172.21.15.59:0/3929361457 conn(0x7fb7c00b35e0 msgr2=0x7fb7c013b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:51.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.743+0000 7fb7c509c640 1 -- 172.21.15.59:0/3929361457 shutdown_connections 2024-03-29T11:25:51.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.745+0000 7fb7c509c640 1 -- 172.21.15.59:0/3929361457 wait complete. 2024-03-29T11:25:51.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:51.745+0000 7fb7c509c640 1 librados: shutdown 2024-03-29T11:25:51.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:58.555 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:58.706 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.719+0000 7fd619bf4640 10 monclient: get_monmap_and_config 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.719+0000 7fd619bf4640 10 monclient: build_initial_monmap 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.719+0000 7fd619bf4640 10 monclient: monmap: 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:25:58.720819+0000 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:25:58.720819+0000 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:25:58.707 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:58.708 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.720+0000 7fd619bf4640 1 Processor -- start 2024-03-29T11:25:58.709 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.721+0000 7fd619bf4640 1 -- start start 2024-03-29T11:25:58.709 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.721+0000 7fd619bf4640 10 monclient: init 2024-03-29T11:25:58.709 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.721+0000 7fd619bf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:58.709 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.721+0000 7fd619bf4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:58.709 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.722+0000 7fd619bf4640 10 monclient(hunting): picked mon.noname-b con 0x7fd6140a9ac0 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:58.709 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.722+0000 7fd619bf4640 10 monclient(hunting): picked mon.noname-a con 0x7fd61413ee90 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:58.709 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.722+0000 7fd619bf4640 10 monclient(hunting): picked mon.noname-c con 0x7fd6140b4100 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:58.710 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.722+0000 7fd619bf4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6140af6f0 con 0x7fd61413ee90 2024-03-29T11:25:58.710 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.722+0000 7fd619bf4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6140aac80 con 0x7fd6140b4100 2024-03-29T11:25:58.710 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.722+0000 7fd619bf4640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6140aaf60 con 0x7fd6140a9ac0 2024-03-29T11:25:58.710 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.722+0000 7fd619bf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:58.710 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.722+0000 7fd619bf4640 10 monclient(hunting): authenticate will time out at 1208.837202s 2024-03-29T11:25:58.711 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.724+0000 7fd6137fe640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fd6140a9ac0 0x7fd6140adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:44290/0 (socket says 172.21.15.59:44290) 2024-03-29T11:25:58.711 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.724+0000 7fd6137fe640 1 -- 172.21.15.59:0/2511518306 learned_addr learned my addr 172.21.15.59:0/2511518306 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:58.711 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.724+0000 7fd6127fc640 1 -- 172.21.15.59:0/2511518306 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618279330 0 0) 0x7fd6140aaf60 con 0x7fd6140a9ac0 2024-03-29T11:25:58.711 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.724+0000 7fd6127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:58.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.724+0000 7fd6127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:58.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.724+0000 7fd6127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:58.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.724+0000 7fd6127fc640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:25:58.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.724+0000 7fd6127fc640 1 -- 172.21.15.59:0/2511518306 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd600003520 con 0x7fd6140a9ac0 2024-03-29T11:25:58.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 1 -- 172.21.15.59:0/2511518306 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4100019492 0 0) 0x7fd600003520 con 0x7fd6140a9ac0 2024-03-29T11:25:58.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 1 -- 172.21.15.59:0/2511518306 >> v1:172.21.15.172:6789/0 conn(0x7fd6140b4100 legacy=0x7fd61414a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 1 -- 172.21.15.59:0/2511518306 >> v1:172.21.15.59:6789/0 conn(0x7fd61413ee90 legacy=0x7fd61413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:58.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 1 monclient: found mon.noname-b 2024-03-29T11:25:58.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:58.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:25:58.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 1 -- 172.21.15.59:0/2511518306 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61413e970 con 0x7fd6140a9ac0 2024-03-29T11:25:58.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:58.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:58.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 1 -- 172.21.15.59:0/2511518306 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd604001a10 con 0x7fd6140a9ac0 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd6127fc640 10 monclient: dump: 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:58.716 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:58.717 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:58.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd619bf4640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:25:58.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.728+0000 7fd619bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:58.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.731+0000 7fd6127fc640 1 -- 172.21.15.59:0/2511518306 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd604001b70 con 0x7fd6140a9ac0 2024-03-29T11:25:58.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.731+0000 7fd6127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:58.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.731+0000 7fd6127fc640 1 -- 172.21.15.59:0/2511518306 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd604001fc0 con 0x7fd6140a9ac0 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.731+0000 7fd6127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.731+0000 7fd6127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.731+0000 7fd6127fc640 10 monclient: dump: 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:58.718 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:58.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.732+0000 7fd619bf4640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:58.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.732+0000 7fd619bf4640 4 set_mon_vals no callback set 2024-03-29T11:25:58.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.733+0000 7fd619bf4640 10 monclient: shutdown 2024-03-29T11:25:58.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.733+0000 7fd619bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:58.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.733+0000 7fd619bf4640 1 -- 172.21.15.59:0/2511518306 >> v1:172.21.15.59:6790/0 conn(0x7fd6140a9ac0 legacy=0x7fd6140adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.722 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.736+0000 7fd619bf4640 1 -- 172.21.15.59:0/2511518306 shutdown_connections 2024-03-29T11:25:58.722 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.736+0000 7fd619bf4640 1 -- 172.21.15.59:0/2511518306 >> 172.21.15.59:0/2511518306 conn(0x7fd6140b35e0 msgr2=0x7fd6140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:58.723 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.736+0000 7fd619bf4640 1 -- 172.21.15.59:0/2511518306 shutdown_connections 2024-03-29T11:25:58.723 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.736+0000 7fd619bf4640 1 -- 172.21.15.59:0/2511518306 wait complete. 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.737+0000 7fd619bf4640 10 monclient: build_initial_monmap 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.737+0000 7fd619bf4640 10 monclient: monmap: 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:25:58.724 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:58.725 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.738+0000 7fd619bf4640 1 librados: starting msgr at 2024-03-29T11:25:58.725 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.738+0000 7fd619bf4640 1 librados: starting objecter 2024-03-29T11:25:58.725 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.738+0000 7fd619bf4640 1 Processor -- start 2024-03-29T11:25:58.725 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.738+0000 7fd619bf4640 1 -- start start 2024-03-29T11:25:58.725 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.738+0000 7fd619bf4640 1 librados: setting wanted keys 2024-03-29T11:25:58.726 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.738+0000 7fd619bf4640 1 librados: calling monclient init 2024-03-29T11:25:58.726 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.738+0000 7fd619bf4640 10 monclient: init 2024-03-29T11:25:58.726 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.739+0000 7fd619bf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:58.726 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.739+0000 7fd619bf4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:58.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.741+0000 7fd619bf4640 10 monclient(hunting): picked mon.noname-a con 0x7fd6140a9ac0 addr v1:172.21.15.59:6789/0 2024-03-29T11:25:58.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.744+0000 7fd619bf4640 10 monclient(hunting): picked mon.noname-b con 0x7fd6140b4100 addr v1:172.21.15.172:6789/0 2024-03-29T11:25:58.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.744+0000 7fd619bf4640 10 monclient(hunting): picked mon.noname-c con 0x7fd61413ee90 addr v1:172.21.15.59:6790/0 2024-03-29T11:25:58.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.745+0000 7fd619bf4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6140726c0 con 0x7fd6140a9ac0 2024-03-29T11:25:58.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.745+0000 7fd619bf4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd614071260 con 0x7fd6140b4100 2024-03-29T11:25:58.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.745+0000 7fd619bf4640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd614071540 con 0x7fd61413ee90 2024-03-29T11:25:58.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.745+0000 7fd619bf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:58.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.745+0000 7fd619bf4640 10 monclient(hunting): authenticate will time out at 1208.859989s 2024-03-29T11:25:58.732 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.745+0000 7fd6137fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd6140a9ac0 0x7fd614071fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58055/0 (socket says 172.21.15.59:58055) 2024-03-29T11:25:58.732 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.745+0000 7fd6137fe640 1 -- 172.21.15.59:0/3204465466 learned_addr learned my addr 172.21.15.59:0/3204465466 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:25:58.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 client.?.objecter ms_handle_connect 0x7fd6140a9ac0 2024-03-29T11:25:58.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:58.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 client.?.objecter ms_handle_connect 0x7fd61413ee90 2024-03-29T11:25:58.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:58.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 client.?.objecter ms_handle_connect 0x7fd6140b4100 2024-03-29T11:25:58.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:58.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619095005 0 0) 0x7fd6140726c0 con 0x7fd6140a9ac0 2024-03-29T11:25:58.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5f4004eb0 con 0x7fd6140a9ac0 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959484251 0 0) 0x7fd614071540 con 0x7fd61413ee90 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.747+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6140726c0 con 0x7fd61413ee90 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691326675 0 0) 0x7fd614071260 con 0x7fd6140b4100 2024-03-29T11:25:58.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:58.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:58.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:58.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:25:58.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd614071540 con 0x7fd6140b4100 2024-03-29T11:25:58.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3011442085 0 0) 0x7fd6140726c0 con 0x7fd61413ee90 2024-03-29T11:25:58.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd614071260 con 0x7fd61413ee90 2024-03-29T11:25:58.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd5fc001760 con 0x7fd61413ee90 2024-03-29T11:25:58.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:58.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): dump: 2024-03-29T11:25:58.739 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:58.739 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:58.739 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.739 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.739 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.739 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3688602443 0 0) 0x7fd5f4004eb0 con 0x7fd6140a9ac0 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5f4006c20 con 0x7fd6140a9ac0 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3974987438 0 0) 0x7fd614071540 con 0x7fd6140b4100 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5f4004eb0 con 0x7fd6140b4100 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd60c001690 con 0x7fd6140b4100 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): dump: 2024-03-29T11:25:58.740 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd604001f40 con 0x7fd6140a9ac0 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.748+0000 7fd60bfff640 10 monclient(hunting): dump: 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:58.741 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1551045820 0 0) 0x7fd5f4006c20 con 0x7fd6140a9ac0 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 >> v1:172.21.15.59:6790/0 conn(0x7fd61413ee90 legacy=0x7fd614141120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 >> v1:172.21.15.172:6789/0 conn(0x7fd6140b4100 legacy=0x7fd6141247a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd60bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd60bfff640 1 monclient: found mon.a 2024-03-29T11:25:58.742 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd60bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd60bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd614140c00 con 0x7fd6140a9ac0 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd60bfff640 10 monclient: _finish_auth 0 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd60bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd60bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd619bf4640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd619bf4640 10 monclient: _renew_subs 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd619bf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.749+0000 7fd619bf4640 1 -- 172.21.15.59:0/3204465466 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd614140890 con 0x7fd6140a9ac0 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.750+0000 7fd619bf4640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.750+0000 7fd619bf4640 10 monclient: _renew_subs 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.750+0000 7fd619bf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.750+0000 7fd619bf4640 1 -- 172.21.15.59:0/3204465466 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd614125090 con 0x7fd6140a9ac0 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.750+0000 7fd619bf4640 1 librados: init done 2024-03-29T11:25:58.743 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.751+0000 7fd619bf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.751+0000 7fd619bf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.751+0000 7fd619bf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.751+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd604004f70 con 0x7fd6140a9ac0 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.751+0000 7fd619bf4640 1 -- 172.21.15.59:0/3204465466 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd614075f10 con 0x7fd6140a9ac0 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.751+0000 7fd60bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.751+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd604005380 con 0x7fd6140a9ac0 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.751+0000 7fd60bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.751+0000 7fd60bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.751+0000 7fd60bfff640 10 monclient: dump: 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.744 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.751+0000 7fd6117fa640 4 set_mon_vals no callback set 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.752+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fd6040055a0 con 0x7fd6140a9ac0 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.753+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fd604085000 con 0x7fd6140a9ac0 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.753+0000 7fd60bfff640 10 client.4464.objecter ms_dispatch 0x7fd614000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.753+0000 7fd60bfff640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.753+0000 7fd60bfff640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.753+0000 7fd60bfff640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-29T11:25:58.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.753+0000 7fd612ffd640 10 monclient: get_auth_request con 0x7fd5f4072ac0 auth_method 0 2024-03-29T11:25:58.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.756+0000 7fd60bfff640 10 client.4464.objecter ms_handle_connect 0x7fd5f4072ac0 2024-03-29T11:25:58.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.762+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd6040508e0 con 0x7fd6140a9ac0 2024-03-29T11:25:58.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.762+0000 7fd60bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:58.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:58.762+0000 7fd60bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:59.087 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.101+0000 7fd619bf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.088 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.101+0000 7fd619bf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.088 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.101+0000 7fd619bf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:25:59.088 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.101+0000 7fd619bf4640 1 -- 172.21.15.59:0/3204465466 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd614140a20 con 0x7fd6140a9ac0 2024-03-29T11:25:59.088 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.103+0000 7fd60bfff640 1 -- 172.21.15.59:0/3204465466 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fd604052810 con 0x7fd6140a9ac0 2024-03-29T11:25:59.089 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.103+0000 7fd60bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.089 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.103+0000 7fd60bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:59.089 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:25:59.089 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.106+0000 7fd619bf4640 10 librados: watch_flush enter 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 10 librados: watch_flush exit 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 10 client.4464.objecter successfully canceled tick 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 1 -- 172.21.15.59:0/3204465466 >> v1:172.21.15.172:6816/2420742773 conn(0x7fd5f4072ac0 legacy=0x7fd5f4074f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 10 monclient: shutdown 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 1 -- 172.21.15.59:0/3204465466 >> v1:172.21.15.59:6789/0 conn(0x7fd6140a9ac0 legacy=0x7fd614071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 1 -- 172.21.15.59:0/3204465466 shutdown_connections 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 1 -- 172.21.15.59:0/3204465466 >> 172.21.15.59:0/3204465466 conn(0x7fd6140b35e0 msgr2=0x7fd6140b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 1 -- 172.21.15.59:0/3204465466 shutdown_connections 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 1 -- 172.21.15.59:0/3204465466 wait complete. 2024-03-29T11:25:59.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:25:59.107+0000 7fd619bf4640 1 librados: shutdown 2024-03-29T11:25:59.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:05.918 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:06.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.071+0000 7fa8f207b640 10 monclient: get_monmap_and_config 2024-03-29T11:26:06.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.071+0000 7fa8f207b640 10 monclient: build_initial_monmap 2024-03-29T11:26:06.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.071+0000 7fa8f207b640 10 monclient: monmap: 2024-03-29T11:26:06.057 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:06.057 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:06.057 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:26:06.072458+0000 2024-03-29T11:26:06.057 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:26:06.072458+0000 2024-03-29T11:26:06.057 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:06.057 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:06.057 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:06.058 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:26:06.058 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:26:06.058 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.072+0000 7fa8f207b640 1 Processor -- start 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.072+0000 7fa8f207b640 1 -- start start 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.072+0000 7fa8f207b640 10 monclient: init 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8f207b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8f207b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8f207b640 10 monclient(hunting): picked mon.noname-b con 0x7fa8ec1422b0 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8f207b640 10 monclient(hunting): picked mon.noname-c con 0x7fa8ec142690 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8f207b640 10 monclient(hunting): picked mon.noname-a con 0x7fa8ec0b17d0 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8f207b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec142c80 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8f207b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec0aae40 con 0x7fa8ec142690 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8f207b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec13fbd0 con 0x7fa8ec1422b0 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8f207b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8f207b640 10 monclient(hunting): authenticate will time out at 1216.187382s 2024-03-29T11:26:06.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8eb7fe640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fa8ec1422b0 0x7fa8ec13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:36800/0 (socket says 172.21.15.59:36800) 2024-03-29T11:26:06.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.073+0000 7fa8eb7fe640 1 -- 172.21.15.59:0/1703118839 learned_addr learned my addr 172.21.15.59:0/1703118839 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:06.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.075+0000 7fa8ea7fc640 1 -- 172.21.15.59:0/1703118839 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639979884 0 0) 0x7fa8ec13fbd0 con 0x7fa8ec1422b0 2024-03-29T11:26:06.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.075+0000 7fa8ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.075+0000 7fa8ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.075+0000 7fa8ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.075+0000 7fa8ea7fc640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:26:06.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.075+0000 7fa8ea7fc640 1 -- 172.21.15.59:0/1703118839 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d4003520 con 0x7fa8ec1422b0 2024-03-29T11:26:06.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.075+0000 7fa8ea7fc640 1 -- 172.21.15.59:0/1703118839 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1728829186 0 0) 0x7fa8d4003520 con 0x7fa8ec1422b0 2024-03-29T11:26:06.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.075+0000 7fa8ea7fc640 1 -- 172.21.15.59:0/1703118839 >> v1:172.21.15.172:6789/0 conn(0x7fa8ec142690 legacy=0x7fa8ec0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:06.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.075+0000 7fa8ea7fc640 1 -- 172.21.15.59:0/1703118839 >> v1:172.21.15.59:6789/0 conn(0x7fa8ec0b17d0 legacy=0x7fa8ec0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:06.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 1 monclient: found mon.noname-b 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 1 -- 172.21.15.59:0/1703118839 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec151940 con 0x7fa8ec1422b0 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 1 -- 172.21.15.59:0/1703118839 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa8e4001a10 con 0x7fa8ec1422b0 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 10 monclient: dump: 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:06.062 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8f207b640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8f207b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 1 -- 172.21.15.59:0/1703118839 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8e4001b70 con 0x7fa8ec1422b0 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 1 -- 172.21.15.59:0/1703118839 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa8e4001fc0 con 0x7fa8ec1422b0 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:06.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8ea7fc640 10 monclient: dump: 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8f207b640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:06.064 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.076+0000 7fa8f207b640 4 set_mon_vals no callback set 2024-03-29T11:26:06.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.077+0000 7fa8f207b640 10 monclient: shutdown 2024-03-29T11:26:06.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.077+0000 7fa8f207b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:06.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.077+0000 7fa8f207b640 1 -- 172.21.15.59:0/1703118839 >> v1:172.21.15.59:6790/0 conn(0x7fa8ec1422b0 legacy=0x7fa8ec13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.077+0000 7fa8f207b640 1 -- 172.21.15.59:0/1703118839 shutdown_connections 2024-03-29T11:26:06.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.077+0000 7fa8f207b640 1 -- 172.21.15.59:0/1703118839 >> 172.21.15.59:0/1703118839 conn(0x7fa8ec066a00 msgr2=0x7fa8ec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.077+0000 7fa8f207b640 1 -- 172.21.15.59:0/1703118839 shutdown_connections 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.077+0000 7fa8f207b640 1 -- 172.21.15.59:0/1703118839 wait complete. 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.078+0000 7fa8f207b640 10 monclient: build_initial_monmap 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.078+0000 7fa8f207b640 10 monclient: monmap: 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:26:06.072 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.078+0000 7fa8f207b640 1 librados: starting msgr at 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.078+0000 7fa8f207b640 1 librados: starting objecter 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.079+0000 7fa8f207b640 1 Processor -- start 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.079+0000 7fa8f207b640 1 -- start start 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.079+0000 7fa8f207b640 1 librados: setting wanted keys 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.079+0000 7fa8f207b640 1 librados: calling monclient init 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.079+0000 7fa8f207b640 10 monclient: init 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.080+0000 7fa8f207b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.080+0000 7fa8f207b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.080+0000 7fa8f207b640 10 monclient(hunting): picked mon.noname-c con 0x7fa8ec0b17d0 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.080+0000 7fa8f207b640 10 monclient(hunting): picked mon.noname-b con 0x7fa8ec1422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.080+0000 7fa8f207b640 10 monclient(hunting): picked mon.noname-a con 0x7fa8ec142690 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.080+0000 7fa8f207b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec12f4d0 con 0x7fa8ec142690 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.080+0000 7fa8f207b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec06e910 con 0x7fa8ec1422b0 2024-03-29T11:26:06.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.080+0000 7fa8f207b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec06ebf0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.080+0000 7fa8f207b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.080+0000 7fa8f207b640 10 monclient(hunting): authenticate will time out at 1216.193947s 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.081+0000 7fa8eaffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa8ec1422b0 0x7fa8ec12e6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:52848/0 (socket says 172.21.15.59:52848) 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.081+0000 7fa8eaffd640 1 -- 172.21.15.59:0/1599759557 learned_addr learned my addr 172.21.15.59:0/1599759557 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.081+0000 7fa8d3fff640 10 client.?.objecter ms_handle_connect 0x7fa8ec0b17d0 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.081+0000 7fa8d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.081+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286343427 0 0) 0x7fa8ec06ebf0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.081+0000 7fa8d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.081+0000 7fa8d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.081+0000 7fa8d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.081+0000 7fa8d3fff640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.081+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d8004eb0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.074 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.082+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1377450017 0 0) 0x7fa8d8004eb0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.082+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8ec06ebf0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.082+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa8e4001e80 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.082+0000 7fa8d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.082+0000 7fa8d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.082+0000 7fa8d3fff640 10 monclient(hunting): dump: 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:06.075 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.082+0000 7fa8d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.082+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 874358213 0 0) 0x7fa8ec06ebf0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.082+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 >> v1:172.21.15.172:6789/0 conn(0x7fa8ec1422b0 legacy=0x7fa8ec12e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.082+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 >> v1:172.21.15.59:6789/0 conn(0x7fa8ec142690 legacy=0x7fa8ec12edc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8d3fff640 1 monclient: found mon.c 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8d3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec06eed0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8f207b640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8f207b640 10 monclient: _renew_subs 2024-03-29T11:26:06.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8f207b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8f207b640 1 -- 172.21.15.59:0/1599759557 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ec131ad0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8f207b640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8f207b640 10 monclient: _renew_subs 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8f207b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8f207b640 1 -- 172.21.15.59:0/1599759557 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ec132720 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.083+0000 7fa8f207b640 1 librados: init done 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.084+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8e4001fe0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.084+0000 7fa8d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.084+0000 7fa8e9ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.085+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa8e4005110 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.086+0000 7fa8d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.086+0000 7fa8d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.086+0000 7fa8d3fff640 10 monclient: dump: 2024-03-29T11:26:06.077 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.086+0000 7fa8f207b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.086+0000 7fa8f207b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.086+0000 7fa8f207b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.086+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fa8e401b0e0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.078 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.086+0000 7fa8f207b640 1 -- 172.21.15.59:0/1599759557 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc005080 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.080 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.087+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fa8e4084fa0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.080 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.087+0000 7fa8d3fff640 10 client.4469.objecter ms_dispatch 0x7fa8ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:06.080 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.087+0000 7fa8d3fff640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:06.080 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.087+0000 7fa8d3fff640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:06.080 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.087+0000 7fa8eaffd640 10 monclient: get_auth_request con 0x7fa8d806dc20 auth_method 0 2024-03-29T11:26:06.080 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.087+0000 7fa8d3fff640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T11:26:06.080 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.087+0000 7fa8d3fff640 10 client.4469.objecter ms_handle_connect 0x7fa8d806dc20 2024-03-29T11:26:06.080 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.094+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8e40506f0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.080 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.094+0000 7fa8d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.080 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.094+0000 7fa8d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:06.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.419+0000 7fa8f207b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.419+0000 7fa8f207b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.419+0000 7fa8f207b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:06.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.419+0000 7fa8f207b640 1 -- 172.21.15.59:0/1599759557 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8cc005330 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.420+0000 7fa8d3fff640 1 -- 172.21.15.59:0/1599759557 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fa8e40527d0 con 0x7fa8ec0b17d0 2024-03-29T11:26:06.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.420+0000 7fa8d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.420+0000 7fa8d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:06.408 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:26:06.408 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:06.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.425+0000 7fa8f207b640 10 librados: watch_flush enter 2024-03-29T11:26:06.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.425+0000 7fa8f207b640 10 librados: watch_flush exit 2024-03-29T11:26:06.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.425+0000 7fa8f207b640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T11:26:06.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.425+0000 7fa8f207b640 10 client.4469.objecter successfully canceled tick 2024-03-29T11:26:06.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.425+0000 7fa8f207b640 1 -- 172.21.15.59:0/1599759557 >> v1:172.21.15.172:6816/2420742773 conn(0x7fa8d806dc20 legacy=0x7fa8d80700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.425+0000 7fa8f207b640 10 monclient: shutdown 2024-03-29T11:26:06.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.425+0000 7fa8f207b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:06.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.425+0000 7fa8f207b640 1 -- 172.21.15.59:0/1599759557 >> v1:172.21.15.59:6790/0 conn(0x7fa8ec0b17d0 legacy=0x7fa8ec132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.425+0000 7fa8f207b640 1 -- 172.21.15.59:0/1599759557 shutdown_connections 2024-03-29T11:26:06.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.425+0000 7fa8f207b640 1 -- 172.21.15.59:0/1599759557 >> 172.21.15.59:0/1599759557 conn(0x7fa8ec066a00 msgr2=0x7fa8ec0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:06.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.426+0000 7fa8f207b640 1 -- 172.21.15.59:0/1599759557 shutdown_connections 2024-03-29T11:26:06.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.426+0000 7fa8f207b640 1 -- 172.21.15.59:0/1599759557 wait complete. 2024-03-29T11:26:06.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:06.426+0000 7fa8f207b640 1 librados: shutdown 2024-03-29T11:26:06.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:13.235 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:13.402 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.416+0000 7f99901d6640 10 monclient: get_monmap_and_config 2024-03-29T11:26:13.402 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.416+0000 7f99901d6640 10 monclient: build_initial_monmap 2024-03-29T11:26:13.402 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.416+0000 7f99901d6640 10 monclient: monmap: 2024-03-29T11:26:13.403 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:13.403 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:13.403 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:26:13.417571+0000 2024-03-29T11:26:13.403 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:26:13.417571+0000 2024-03-29T11:26:13.403 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:13.403 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:13.403 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:13.403 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:26:13.403 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:26:13.403 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:13.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.418+0000 7f99901d6640 1 Processor -- start 2024-03-29T11:26:13.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.418+0000 7f99901d6640 1 -- start start 2024-03-29T11:26:13.404 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.418+0000 7f99901d6640 10 monclient: init 2024-03-29T11:26:13.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.421+0000 7f99901d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:13.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.421+0000 7f99901d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:13.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.421+0000 7f99901d6640 10 monclient(hunting): picked mon.noname-c con 0x7f99880708a0 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:13.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.421+0000 7f99901d6640 10 monclient(hunting): picked mon.noname-b con 0x7f998806fe90 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:13.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.421+0000 7f99901d6640 10 monclient(hunting): picked mon.noname-a con 0x7f9988143c50 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:13.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.421+0000 7f99901d6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9988072630 con 0x7f9988143c50 2024-03-29T11:26:13.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.421+0000 7f99901d6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9988071410 con 0x7f99880708a0 2024-03-29T11:26:13.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.421+0000 7f99901d6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99880afaa0 con 0x7f998806fe90 2024-03-29T11:26:13.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.421+0000 7f99901d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:13.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.421+0000 7f99901d6640 10 monclient(hunting): authenticate will time out at 1223.535426s 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.422+0000 7f998df72640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f99880708a0 0x7f99880af390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:51444/0 (socket says 172.21.15.59:51444) 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.422+0000 7f998df72640 1 -- 172.21.15.59:0/4093950072 learned_addr learned my addr 172.21.15.59:0/4093950072 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.422+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905629334 0 0) 0x7f99880afaa0 con 0x7f998806fe90 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.422+0000 7f998cf70640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.422+0000 7f998cf70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.422+0000 7f998cf70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.422+0000 7f998cf70640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.422+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9974003520 con 0x7f998806fe90 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512582545 0 0) 0x7f9988071410 con 0x7f99880708a0 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:26:13.409 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99880afaa0 con 0x7f99880708a0 2024-03-29T11:26:13.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888618268 0 0) 0x7f9988072630 con 0x7f9988143c50 2024-03-29T11:26:13.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:26:13.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9988071410 con 0x7f9988143c50 2024-03-29T11:26:13.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3003028269 0 0) 0x7f99880afaa0 con 0x7f99880708a0 2024-03-29T11:26:13.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 >> v1:172.21.15.59:6790/0 conn(0x7f998806fe90 legacy=0x7f9988143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 >> v1:172.21.15.59:6789/0 conn(0x7f9988143c50 legacy=0x7f998814b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.411 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient: _finish_hunting 0 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 1 monclient: found mon.noname-c 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99880ac8b0 con 0x7f99880708a0 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient: _finish_auth 0 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient: _check_auth_tickets 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f99800019f0 con 0x7f99880708a0 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f998cf70640 10 monclient: dump: 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:13.412 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f99901d6640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.423+0000 7f99901d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.424+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9980001b50 con 0x7f99880708a0 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.424+0000 7f998cf70640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.424+0000 7f998cf70640 1 -- 172.21.15.59:0/4093950072 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9980001f60 con 0x7f99880708a0 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.424+0000 7f998cf70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.424+0000 7f998cf70640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.424+0000 7f998cf70640 10 monclient: dump: 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.424+0000 7f99901d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.424+0000 7f99901d6640 4 set_mon_vals no callback set 2024-03-29T11:26:13.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.425+0000 7f99901d6640 10 monclient: shutdown 2024-03-29T11:26:13.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.425+0000 7f99901d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:13.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.425+0000 7f99901d6640 1 -- 172.21.15.59:0/4093950072 >> v1:172.21.15.172:6789/0 conn(0x7f99880708a0 legacy=0x7f99880af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.425+0000 7f99901d6640 1 -- 172.21.15.59:0/4093950072 shutdown_connections 2024-03-29T11:26:13.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.425+0000 7f99901d6640 1 -- 172.21.15.59:0/4093950072 >> 172.21.15.59:0/4093950072 conn(0x7f99880675b0 msgr2=0x7f99880b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:13.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.425+0000 7f99901d6640 1 -- 172.21.15.59:0/4093950072 shutdown_connections 2024-03-29T11:26:13.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.425+0000 7f99901d6640 1 -- 172.21.15.59:0/4093950072 wait complete. 2024-03-29T11:26:13.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.426+0000 7f99901d6640 10 monclient: build_initial_monmap 2024-03-29T11:26:13.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.427+0000 7f99901d6640 10 monclient: monmap: 2024-03-29T11:26:13.423 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:13.423 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:13.423 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:26:13.423 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.427+0000 7f99901d6640 1 librados: starting msgr at 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.427+0000 7f99901d6640 1 librados: starting objecter 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.428+0000 7f99901d6640 1 Processor -- start 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.428+0000 7f99901d6640 1 -- start start 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.428+0000 7f99901d6640 1 librados: setting wanted keys 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.428+0000 7f99901d6640 1 librados: calling monclient init 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.428+0000 7f99901d6640 10 monclient: init 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f99901d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:13.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f99901d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f99901d6640 10 monclient(hunting): picked mon.noname-a con 0x7f998806fe90 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f99901d6640 10 monclient(hunting): picked mon.noname-c con 0x7f99880708a0 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f99901d6640 10 monclient(hunting): picked mon.noname-b con 0x7f9988143c50 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f99901d6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9988139aa0 con 0x7f998806fe90 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f99901d6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9988136ab0 con 0x7f9988143c50 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f99901d6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9988139cd0 con 0x7f99880708a0 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f99901d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f99901d6640 10 monclient(hunting): authenticate will time out at 1223.542925s 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f998df72640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f998806fe90 0x7f998813a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34172/0 (socket says 172.21.15.59:34172) 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f998df72640 1 -- 172.21.15.59:0/1964308069 learned_addr learned my addr 172.21.15.59:0/1964308069 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f997a7fc640 10 client.?.objecter ms_handle_connect 0x7f998806fe90 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.429+0000 7f997a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 client.?.objecter ms_handle_connect 0x7f99880708a0 2024-03-29T11:26:13.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:13.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 client.?.objecter ms_handle_connect 0x7f9988143c50 2024-03-29T11:26:13.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:13.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567273741 0 0) 0x7f9988072630 con 0x7f9988143c50 2024-03-29T11:26:13.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:26:13.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f996c004eb0 con 0x7f9988143c50 2024-03-29T11:26:13.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792561652 0 0) 0x7f9988139cd0 con 0x7f99880708a0 2024-03-29T11:26:13.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:26:13.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9988072630 con 0x7f99880708a0 2024-03-29T11:26:13.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176566170 0 0) 0x7f9988139aa0 con 0x7f998806fe90 2024-03-29T11:26:13.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:26:13.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.430+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9988139cd0 con 0x7f998806fe90 2024-03-29T11:26:13.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.431+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3305725084 0 0) 0x7f996c004eb0 con 0x7f9988143c50 2024-03-29T11:26:13.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.431+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9988139aa0 con 0x7f9988143c50 2024-03-29T11:26:13.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.431+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9984001790 con 0x7f9988143c50 2024-03-29T11:26:13.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.431+0000 7f997a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.431+0000 7f997a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:13.429 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.431+0000 7f997a7fc640 10 monclient(hunting): dump: 2024-03-29T11:26:13.429 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:13.429 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:13.430 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.430 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.430 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.430 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:13.430 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:13.430 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:13.430 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:13.431 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:13.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.431+0000 7f997a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:13.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.431+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3124253194 0 0) 0x7f9988072630 con 0x7f99880708a0 2024-03-29T11:26:13.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.431+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f996c006b80 con 0x7f99880708a0 2024-03-29T11:26:13.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.431+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f997c001770 con 0x7f99880708a0 2024-03-29T11:26:13.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:13.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 10 monclient(hunting): dump: 2024-03-29T11:26:13.432 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:13.432 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:13.432 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.432 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.432 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.432 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:13.432 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:13.433 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:13.433 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:13.433 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:13.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 758833174 0 0) 0x7f9988139cd0 con 0x7f998806fe90 2024-03-29T11:26:13.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9988072630 con 0x7f998806fe90 2024-03-29T11:26:13.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9980001f20 con 0x7f998806fe90 2024-03-29T11:26:13.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:13.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 10 monclient(hunting): dump: 2024-03-29T11:26:13.434 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:13.434 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:13.434 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.435 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.435 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.435 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:13.435 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:13.435 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:13.435 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:13.435 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:13.435 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2340633716 0 0) 0x7f996c006b80 con 0x7f99880708a0 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 >> v1:172.21.15.172:6789/0 conn(0x7f9988143c50 legacy=0x7f99881e04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 >> v1:172.21.15.59:6789/0 conn(0x7f998806fe90 legacy=0x7f998813a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 1 monclient: found mon.c 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9988139fb0 con 0x7f99880708a0 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f997a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.432+0000 7f99901d6640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f99901d6640 10 monclient: _renew_subs 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f99901d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f99901d6640 1 -- 172.21.15.59:0/1964308069 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99880ab720 con 0x7f99880708a0 2024-03-29T11:26:13.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f99901d6640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f99901d6640 10 monclient: _renew_subs 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f99901d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f99901d6640 1 -- 172.21.15.59:0/1964308069 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99880ac560 con 0x7f99880708a0 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f99901d6640 1 librados: init done 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f997c001c60 con 0x7f99880708a0 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f997a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f997c003790 con 0x7f99880708a0 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f997a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f997a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f997a7fc640 10 monclient: dump: 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.437 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.433+0000 7f997b7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.435+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f997c019760 con 0x7f99880708a0 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.436+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f997c0836b0 con 0x7f99880708a0 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.436+0000 7f997a7fc640 10 client.4475.objecter ms_dispatch 0x7f9988000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.436+0000 7f997a7fc640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.436+0000 7f997a7fc640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:13.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.436+0000 7f997a7fc640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:26:13.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.436+0000 7f998df72640 10 monclient: get_auth_request con 0x7f996c06e500 auth_method 0 2024-03-29T11:26:13.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.437+0000 7f99901d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.437+0000 7f99901d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.437+0000 7f99901d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:13.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.437+0000 7f99901d6640 1 -- 172.21.15.59:0/1964308069 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9970005080 con 0x7f99880708a0 2024-03-29T11:26:13.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.437+0000 7f997a7fc640 10 client.4475.objecter ms_handle_connect 0x7f996c06e500 2024-03-29T11:26:13.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.452+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f997c04ef90 con 0x7f99880708a0 2024-03-29T11:26:13.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.452+0000 7f997a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.452+0000 7f997a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:13.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.841+0000 7f99901d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:13.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.841+0000 7f99901d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:13.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.841+0000 7f99901d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:13.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.841+0000 7f99901d6640 1 -- 172.21.15.59:0/1964308069 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9970005330 con 0x7f99880708a0 2024-03-29T11:26:13.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.843+0000 7f997a7fc640 1 -- 172.21.15.59:0/1964308069 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f997c050ec0 con 0x7f99880708a0 2024-03-29T11:26:13.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.843+0000 7f997a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:13.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.843+0000 7f997a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:13.828 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:26:13.828 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:13.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.846+0000 7f99901d6640 10 librados: watch_flush enter 2024-03-29T11:26:13.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.846+0000 7f99901d6640 10 librados: watch_flush exit 2024-03-29T11:26:13.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.846+0000 7f99901d6640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:26:13.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.846+0000 7f99901d6640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:26:13.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.846+0000 7f99901d6640 1 -- 172.21.15.59:0/1964308069 >> v1:172.21.15.172:6816/2420742773 conn(0x7f996c06e500 legacy=0x7f996c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.846+0000 7f99901d6640 10 monclient: shutdown 2024-03-29T11:26:13.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.846+0000 7f99901d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:13.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.846+0000 7f99901d6640 1 -- 172.21.15.59:0/1964308069 >> v1:172.21.15.59:6790/0 conn(0x7f99880708a0 legacy=0x7f99880aa5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.833 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.847+0000 7f99901d6640 1 -- 172.21.15.59:0/1964308069 shutdown_connections 2024-03-29T11:26:13.833 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.847+0000 7f99901d6640 1 -- 172.21.15.59:0/1964308069 >> 172.21.15.59:0/1964308069 conn(0x7f99880675b0 msgr2=0x7f9988137880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:13.833 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.847+0000 7f99901d6640 1 -- 172.21.15.59:0/1964308069 shutdown_connections 2024-03-29T11:26:13.833 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.847+0000 7f99901d6640 1 -- 172.21.15.59:0/1964308069 wait complete. 2024-03-29T11:26:13.833 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:13.847+0000 7f99901d6640 1 librados: shutdown 2024-03-29T11:26:13.850 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:20.658 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:20.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.823+0000 7f91d71e6640 10 monclient: get_monmap_and_config 2024-03-29T11:26:20.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.823+0000 7f91d71e6640 10 monclient: build_initial_monmap 2024-03-29T11:26:20.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.823+0000 7f91d71e6640 10 monclient: monmap: 2024-03-29T11:26:20.810 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:20.810 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:20.810 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:26:20.824630+0000 2024-03-29T11:26:20.810 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:26:20.824630+0000 2024-03-29T11:26:20.810 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:20.810 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:20.810 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:20.810 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:26:20.810 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:26:20.811 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:20.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.825+0000 7f91d71e6640 1 Processor -- start 2024-03-29T11:26:20.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.825+0000 7f91d71e6640 1 -- start start 2024-03-29T11:26:20.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.825+0000 7f91d71e6640 10 monclient: init 2024-03-29T11:26:20.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.826+0000 7f91d71e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:20.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.826+0000 7f91d71e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:20.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.826+0000 7f91d71e6640 10 monclient(hunting): picked mon.noname-c con 0x7f91d00b46b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:20.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.826+0000 7f91d71e6640 10 monclient(hunting): picked mon.noname-b con 0x7f91d00aeae0 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:20.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.826+0000 7f91d71e6640 10 monclient(hunting): picked mon.noname-a con 0x7f91d013fa10 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:20.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.826+0000 7f91d71e6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d00b06a0 con 0x7f91d013fa10 2024-03-29T11:26:20.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.827+0000 7f91d4f82640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f91d00b46b0 0x7f91d00b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:60122/0 (socket says 172.21.15.59:60122) 2024-03-29T11:26:20.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.827+0000 7f91d4f82640 1 -- 172.21.15.59:0/1486081866 learned_addr learned my addr 172.21.15.59:0/1486081866 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:20.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.827+0000 7f91d71e6640 1 -- 172.21.15.59:0/1486081866 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d00b4230 con 0x7f91d00b46b0 2024-03-29T11:26:20.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.827+0000 7f91d71e6640 1 -- 172.21.15.59:0/1486081866 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d00b0910 con 0x7f91d00aeae0 2024-03-29T11:26:20.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.827+0000 7f91d71e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:20.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.827+0000 7f91d71e6640 10 monclient(hunting): authenticate will time out at 1230.941171s 2024-03-29T11:26:20.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626736550 0 0) 0x7f91d00b0910 con 0x7f91d00aeae0 2024-03-29T11:26:20.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:26:20.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c0003520 con 0x7f91d00aeae0 2024-03-29T11:26:20.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369656530 0 0) 0x7f91d00b06a0 con 0x7f91d013fa10 2024-03-29T11:26:20.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:26:20.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d00b0910 con 0x7f91d013fa10 2024-03-29T11:26:20.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848629836 0 0) 0x7f91d00b4230 con 0x7f91d00b46b0 2024-03-29T11:26:20.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:26:20.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d00b06a0 con 0x7f91d00b46b0 2024-03-29T11:26:20.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2422518137 0 0) 0x7f91c0003520 con 0x7f91d00aeae0 2024-03-29T11:26:20.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 >> v1:172.21.15.172:6789/0 conn(0x7f91d00b46b0 legacy=0x7f91d00b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 >> v1:172.21.15.59:6789/0 conn(0x7f91d013fa10 legacy=0x7f91d014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 monclient: found mon.noname-b 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d00af6d0 con 0x7f91d00aeae0 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91d71e6640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91d71e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f91bc001690 con 0x7f91d00aeae0 2024-03-29T11:26:20.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 10 monclient: dump: 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:20.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91d71e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.828+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91bc0018e0 con 0x7f91d00aeae0 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.829+0000 7f91cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.829+0000 7f91cf7fe640 1 -- 172.21.15.59:0/1486081866 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f91bc001c70 con 0x7f91d00aeae0 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.829+0000 7f91cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.829+0000 7f91cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.829+0000 7f91cf7fe640 10 monclient: dump: 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.820 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:20.821 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:20.821 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:20.821 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:20.821 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:20.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.829+0000 7f91d71e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:20.821 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.829+0000 7f91d71e6640 4 set_mon_vals no callback set 2024-03-29T11:26:20.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.830+0000 7f91d71e6640 10 monclient: shutdown 2024-03-29T11:26:20.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.830+0000 7f91d71e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:20.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.830+0000 7f91d71e6640 1 -- 172.21.15.59:0/1486081866 >> v1:172.21.15.59:6790/0 conn(0x7f91d00aeae0 legacy=0x7f91d013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.830+0000 7f91d71e6640 1 -- 172.21.15.59:0/1486081866 shutdown_connections 2024-03-29T11:26:20.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.830+0000 7f91d71e6640 1 -- 172.21.15.59:0/1486081866 >> 172.21.15.59:0/1486081866 conn(0x7f91d0066a00 msgr2=0x7f91d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:20.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.830+0000 7f91d71e6640 1 -- 172.21.15.59:0/1486081866 shutdown_connections 2024-03-29T11:26:20.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.830+0000 7f91d71e6640 1 -- 172.21.15.59:0/1486081866 wait complete. 2024-03-29T11:26:20.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.831+0000 7f91d71e6640 10 monclient: build_initial_monmap 2024-03-29T11:26:20.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.831+0000 7f91d71e6640 10 monclient: monmap: 2024-03-29T11:26:20.832 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:20.832 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.832+0000 7f91d71e6640 1 librados: starting msgr at 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.832+0000 7f91d71e6640 1 librados: starting objecter 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.832+0000 7f91d71e6640 1 Processor -- start 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.832+0000 7f91d71e6640 1 -- start start 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.832+0000 7f91d71e6640 1 librados: setting wanted keys 2024-03-29T11:26:20.833 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.832+0000 7f91d71e6640 1 librados: calling monclient init 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.832+0000 7f91d71e6640 10 monclient: init 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.833+0000 7f91d71e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.833+0000 7f91d71e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91d71e6640 10 monclient(hunting): picked mon.noname-a con 0x7f91d00aeae0 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91d71e6640 10 monclient(hunting): picked mon.noname-b con 0x7f91d00b46b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91d4f82640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f91d00aeae0 0x7f91d01db970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:52100/0 (socket says 172.21.15.59:52100) 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91d4f82640 1 -- 172.21.15.59:0/1788953452 learned_addr learned my addr 172.21.15.59:0/1788953452 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91d71e6640 10 monclient(hunting): picked mon.noname-c con 0x7f91d013fa10 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d01dc4f0 con 0x7f91d00aeae0 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d01dc080 con 0x7f91d00b46b0 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91cd7fa640 10 client.?.objecter ms_handle_connect 0x7f91d00aeae0 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91cd7fa640 10 client.?.objecter ms_handle_connect 0x7f91d00b46b0 2024-03-29T11:26:20.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056270398 0 0) 0x7f91d01dc4f0 con 0x7f91d00aeae0 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d01dc210 con 0x7f91d013fa10 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91d71e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.834+0000 7f91d71e6640 10 monclient(hunting): authenticate will time out at 1230.948653s 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b4004eb0 con 0x7f91d00aeae0 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 client.?.objecter ms_handle_connect 0x7f91d013fa10 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714557391 0 0) 0x7f91d01dc080 con 0x7f91d00b46b0 2024-03-29T11:26:20.835 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.836 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.836 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.836 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:26:20.836 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d01dc4f0 con 0x7f91d00b46b0 2024-03-29T11:26:20.836 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1368455448 0 0) 0x7f91b4004eb0 con 0x7f91d00aeae0 2024-03-29T11:26:20.836 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91d01dc080 con 0x7f91d00aeae0 2024-03-29T11:26:20.836 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f91c40018b0 con 0x7f91d00aeae0 2024-03-29T11:26:20.836 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): dump: 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.835+0000 7f91cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:20.837 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626611116 0 0) 0x7f91d01dc210 con 0x7f91d013fa10 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b4004eb0 con 0x7f91d013fa10 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1903489807 0 0) 0x7f91d01dc080 con 0x7f91d00aeae0 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 >> v1:172.21.15.59:6790/0 conn(0x7f91d013fa10 legacy=0x7f91d01dee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 >> v1:172.21.15.172:6789/0 conn(0x7f91d00b46b0 legacy=0x7f91d01d8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 1 monclient: found mon.a 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d0137790 con 0x7f91d00aeae0 2024-03-29T11:26:20.838 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91d71e6640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91d71e6640 10 monclient: _renew_subs 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91d71e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d01379e0 con 0x7f91d00aeae0 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91d71e6640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91d71e6640 10 monclient: _renew_subs 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91d71e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d01d9390 con 0x7f91d00aeae0 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.836+0000 7f91d71e6640 1 librados: init done 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.837+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91c4001a10 con 0x7f91d00aeae0 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.837+0000 7f91cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:20.839 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.837+0000 7f91ceffd640 4 set_mon_vals no callback set 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.837+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f91c40049c0 con 0x7f91d00aeae0 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.837+0000 7f91cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.837+0000 7f91cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.837+0000 7f91cd7fa640 10 monclient: dump: 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:20.840 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.838+0000 7f91d71e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:20.841 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.838+0000 7f91d71e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:20.841 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.838+0000 7f91d71e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:26:20.841 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.838+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b0005080 con 0x7f91d00aeae0 2024-03-29T11:26:20.841 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.839+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f91c4004b20 con 0x7f91d00aeae0 2024-03-29T11:26:20.841 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.840+0000 7f91cffff640 10 monclient: get_auth_request con 0x7f91b406db70 auth_method 0 2024-03-29T11:26:20.841 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.840+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f91c4084a20 con 0x7f91d00aeae0 2024-03-29T11:26:20.841 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.840+0000 7f91cd7fa640 10 client.4476.objecter ms_dispatch 0x7f91d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:20.841 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.840+0000 7f91cd7fa640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:20.841 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.840+0000 7f91cd7fa640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:20.841 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.840+0000 7f91cd7fa640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-29T11:26:20.841 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.841+0000 7f91cd7fa640 10 client.4476.objecter ms_handle_connect 0x7f91b406db70 2024-03-29T11:26:20.842 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.848+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91c4050360 con 0x7f91d00aeae0 2024-03-29T11:26:20.842 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.849+0000 7f91cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:20.842 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:20.849+0000 7f91cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:21.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.231+0000 7f91d71e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.231+0000 7f91d71e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.231+0000 7f91d71e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:26:21.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.231+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91b0005330 con 0x7f91d00aeae0 2024-03-29T11:26:21.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.232+0000 7f91cd7fa640 1 -- 172.21.15.59:0/1788953452 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f91c4052290 con 0x7f91d00aeae0 2024-03-29T11:26:21.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.232+0000 7f91cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.232+0000 7f91cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:21.218 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:26:21.218 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:21.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.235+0000 7f91d71e6640 10 librados: watch_flush enter 2024-03-29T11:26:21.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.236+0000 7f91d71e6640 10 librados: watch_flush exit 2024-03-29T11:26:21.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.236+0000 7f91d71e6640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-29T11:26:21.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.236+0000 7f91d71e6640 10 client.4476.objecter successfully canceled tick 2024-03-29T11:26:21.222 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.236+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 >> v1:172.21.15.172:6816/2420742773 conn(0x7f91b406db70 legacy=0x7f91b4070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.222 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.236+0000 7f91d71e6640 10 monclient: shutdown 2024-03-29T11:26:21.222 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.236+0000 7f91d71e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:21.222 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.236+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 >> v1:172.21.15.59:6789/0 conn(0x7f91d00aeae0 legacy=0x7f91d01db970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.222 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.236+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 shutdown_connections 2024-03-29T11:26:21.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.236+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 >> 172.21.15.59:0/1788953452 conn(0x7f91d0066a00 msgr2=0x7f91d013f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:21.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.237+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 shutdown_connections 2024-03-29T11:26:21.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.237+0000 7f91d71e6640 1 -- 172.21.15.59:0/1788953452 wait complete. 2024-03-29T11:26:21.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:21.237+0000 7f91d71e6640 1 librados: shutdown 2024-03-29T11:26:21.238 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:28.046 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:28.196 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.209+0000 7f16fb59e640 10 monclient: get_monmap_and_config 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.209+0000 7f16fb59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.209+0000 7f16fb59e640 10 monclient: monmap: 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:26:28.210106+0000 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:26:28.210106+0000 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.210+0000 7f16fb59e640 1 Processor -- start 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.210+0000 7f16fb59e640 1 -- start start 2024-03-29T11:26:28.197 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.210+0000 7f16fb59e640 10 monclient: init 2024-03-29T11:26:28.198 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16fb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:28.198 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16fb59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:28.198 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16fb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f16f4034850 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16fb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f16f40c9550 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16fb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f16f402aac0 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16fb59e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f402c440 con 0x7f16f402aac0 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16fb59e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f402aed0 con 0x7f16f40c9550 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16fb59e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f402b190 con 0x7f16f4034850 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16fb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16fb59e640 10 monclient(hunting): authenticate will time out at 1238.325119s 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16f9d9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f16f40c9550 0x7f16f40cbea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:42066/0 (socket says 172.21.15.59:42066) 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.211+0000 7f16f9d9b640 1 -- 172.21.15.59:0/3179970768 learned_addr learned my addr 172.21.15.59:0/3179970768 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.212+0000 7f16f959a640 1 -- 172.21.15.59:0/3179970768 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094475100 0 0) 0x7f16f402aed0 con 0x7f16f40c9550 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.212+0000 7f16f959a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.212+0000 7f16f959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.212+0000 7f16f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.212+0000 7f16f959a640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:26:28.200 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.212+0000 7f16f959a640 1 -- 172.21.15.59:0/3179970768 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e4003520 con 0x7f16f40c9550 2024-03-29T11:26:28.200 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.212+0000 7f16f959a640 1 -- 172.21.15.59:0/3179970768 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 79064542 0 0) 0x7f16e4003520 con 0x7f16f40c9550 2024-03-29T11:26:28.200 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.212+0000 7f16f959a640 1 -- 172.21.15.59:0/3179970768 >> v1:172.21.15.59:6790/0 conn(0x7f16f4034850 legacy=0x7f16f40c8e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:28.202 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 1 -- 172.21.15.59:0/3179970768 >> v1:172.21.15.59:6789/0 conn(0x7f16f402aac0 legacy=0x7f16f40cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 1 monclient: found mon.noname-c 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 1 -- 172.21.15.59:0/3179970768 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f4033060 con 0x7f16f40c9550 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 10 monclient: _finish_auth 0 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 1 -- 172.21.15.59:0/3179970768 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f16f0001690 con 0x7f16f40c9550 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16f959a640 10 monclient: dump: 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:28.203 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16fb59e640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.215+0000 7f16fb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.216+0000 7f16f959a640 1 -- 172.21.15.59:0/3179970768 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16f0001860 con 0x7f16f40c9550 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.216+0000 7f16f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.216+0000 7f16f959a640 1 -- 172.21.15.59:0/3179970768 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f16f0001cb0 con 0x7f16f40c9550 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.216+0000 7f16f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.216+0000 7f16f959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:28.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.216+0000 7f16f959a640 10 monclient: dump: 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.216+0000 7f16fb59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:28.205 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.216+0000 7f16fb59e640 4 set_mon_vals no callback set 2024-03-29T11:26:28.206 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.217+0000 7f16fb59e640 10 monclient: shutdown 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.217+0000 7f16fb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.217+0000 7f16fb59e640 1 -- 172.21.15.59:0/3179970768 >> v1:172.21.15.172:6789/0 conn(0x7f16f40c9550 legacy=0x7f16f40cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.218+0000 7f16fb59e640 1 -- 172.21.15.59:0/3179970768 shutdown_connections 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.218+0000 7f16fb59e640 1 -- 172.21.15.59:0/3179970768 >> 172.21.15.59:0/3179970768 conn(0x7f16f40058f0 msgr2=0x7f16f4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.218+0000 7f16fb59e640 1 -- 172.21.15.59:0/3179970768 shutdown_connections 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.218+0000 7f16fb59e640 1 -- 172.21.15.59:0/3179970768 wait complete. 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.219+0000 7f16fb59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.219+0000 7f16fb59e640 10 monclient: monmap: 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:28.207 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.220+0000 7f16fb59e640 1 librados: starting msgr at 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.220+0000 7f16fb59e640 1 librados: starting objecter 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.220+0000 7f16fb59e640 1 Processor -- start 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.220+0000 7f16fb59e640 1 -- start start 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.220+0000 7f16fb59e640 1 librados: setting wanted keys 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.220+0000 7f16fb59e640 1 librados: calling monclient init 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.220+0000 7f16fb59e640 10 monclient: init 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.221+0000 7f16fb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.221+0000 7f16fb59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.221+0000 7f16fb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f16f402aac0 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.221+0000 7f16fb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f16f4034850 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:28.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.221+0000 7f16fb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f16f40c9550 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:28.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.221+0000 7f16fb59e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f402cab0 con 0x7f16f402aac0 2024-03-29T11:26:28.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.221+0000 7f16fb59e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f4163fb0 con 0x7f16f4034850 2024-03-29T11:26:28.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.221+0000 7f16fb59e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f402dab0 con 0x7f16f40c9550 2024-03-29T11:26:28.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.221+0000 7f16fb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:28.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.221+0000 7f16fb59e640 10 monclient(hunting): authenticate will time out at 1238.334833s 2024-03-29T11:26:28.209 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.222+0000 7f16f9d9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f16f4034850 0x7f16f4164440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:43102/0 (socket says 172.21.15.59:43102) 2024-03-29T11:26:28.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.222+0000 7f16f9d9b640 1 -- 172.21.15.59:0/2987895849 learned_addr learned my addr 172.21.15.59:0/2987895849 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:28.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.223+0000 7f16eaffd640 10 client.?.objecter ms_handle_connect 0x7f16f4034850 2024-03-29T11:26:28.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.223+0000 7f16eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:28.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.224+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057921716 0 0) 0x7f16f4163fb0 con 0x7f16f4034850 2024-03-29T11:26:28.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.224+0000 7f16eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.224+0000 7f16eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.224+0000 7f16eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.224+0000 7f16eaffd640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:26:28.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.224+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16dc004eb0 con 0x7f16f4034850 2024-03-29T11:26:28.210 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 client.?.objecter ms_handle_connect 0x7f16f402aac0 2024-03-29T11:26:28.211 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 client.?.objecter ms_handle_connect 0x7f16f40c9550 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4085276187 0 0) 0x7f16dc004eb0 con 0x7f16f4034850 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16f4163fb0 con 0x7f16f4034850 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544841046 0 0) 0x7f16fc04fdd0 con 0x7f16f402aac0 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16dc004eb0 con 0x7f16f402aac0 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f16f0001c30 con 0x7f16f4034850 2024-03-29T11:26:28.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:28.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 monclient(hunting): dump: 2024-03-29T11:26:28.215 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:28.215 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:28.215 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:28.215 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:28.215 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.215 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:28.215 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:28.215 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.225+0000 7f16eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 574262293 0 0) 0x7f16dc004eb0 con 0x7f16f402aac0 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16fc04fdd0 con 0x7f16f402aac0 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f16fc068810 con 0x7f16f402aac0 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 10 monclient(hunting): dump: 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:28.216 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 913220733 0 0) 0x7f16f4163fb0 con 0x7f16f4034850 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 >> v1:172.21.15.59:6790/0 conn(0x7f16f40c9550 legacy=0x7f16f4160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 >> v1:172.21.15.59:6789/0 conn(0x7f16f402aac0 legacy=0x7f16f40c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 1 monclient: found mon.b 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40326e0 con 0x7f16f4034850 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:28.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16fb59e640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16fb59e640 10 monclient: _renew_subs 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16fb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16fb59e640 1 -- 172.21.15.59:0/2987895849 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f40c5320 con 0x7f16f4034850 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16fb59e640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16fb59e640 10 monclient: _renew_subs 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16fb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16fb59e640 1 -- 172.21.15.59:0/2987895849 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f4160980 con 0x7f16f4034850 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16fb59e640 1 librados: init done 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16f0004a90 con 0x7f16f4034850 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f16f0004ea0 con 0x7f16f4034850 2024-03-29T11:26:28.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16eaffd640 10 monclient: dump: 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:28.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.226+0000 7f16ebfff640 4 set_mon_vals no callback set 2024-03-29T11:26:28.220 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.228+0000 7f16fb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.228+0000 7f16fb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.228+0000 7f16fb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:26:28.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.228+0000 7f16fb59e640 1 -- 172.21.15.59:0/2987895849 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e0005080 con 0x7f16f4034850 2024-03-29T11:26:28.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.228+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f16f001ae70 con 0x7f16f4034850 2024-03-29T11:26:28.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.229+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f16f0084bb0 con 0x7f16f4034850 2024-03-29T11:26:28.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.229+0000 7f16eaffd640 10 client.4474.objecter ms_dispatch 0x7f16fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:28.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.229+0000 7f16eaffd640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:28.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.229+0000 7f16eaffd640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:28.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.229+0000 7f16eaffd640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T11:26:28.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.230+0000 7f16fa59c640 10 monclient: get_auth_request con 0x7f16dc06dd90 auth_method 0 2024-03-29T11:26:28.221 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.233+0000 7f16eaffd640 10 client.4474.objecter ms_handle_connect 0x7f16dc06dd90 2024-03-29T11:26:28.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.238+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16f0050490 con 0x7f16f4034850 2024-03-29T11:26:28.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.238+0000 7f16eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.238+0000 7f16eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:28.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.578+0000 7f16fb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.565 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.578+0000 7f16fb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.565 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.578+0000 7f16fb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:26:28.565 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.578+0000 7f16fb59e640 1 -- 172.21.15.59:0/2987895849 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16e0005330 con 0x7f16f4034850 2024-03-29T11:26:28.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.581+0000 7f16eaffd640 1 -- 172.21.15.59:0/2987895849 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f16f00523c0 con 0x7f16f4034850 2024-03-29T11:26:28.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.581+0000 7f16eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.581+0000 7f16eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:28.567 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:26:28.567 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:28.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.586+0000 7f16fb59e640 10 librados: watch_flush enter 2024-03-29T11:26:28.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.586+0000 7f16fb59e640 10 librados: watch_flush exit 2024-03-29T11:26:28.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.586+0000 7f16fb59e640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T11:26:28.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.586+0000 7f16fb59e640 10 client.4474.objecter successfully canceled tick 2024-03-29T11:26:28.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.586+0000 7f16fb59e640 1 -- 172.21.15.59:0/2987895849 >> v1:172.21.15.172:6816/2420742773 conn(0x7f16dc06dd90 legacy=0x7f16dc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.586+0000 7f16fb59e640 10 monclient: shutdown 2024-03-29T11:26:28.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.586+0000 7f16fb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:28.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.586+0000 7f16fb59e640 1 -- 172.21.15.59:0/2987895849 >> v1:172.21.15.172:6789/0 conn(0x7f16f4034850 legacy=0x7f16f4164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.573 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.587+0000 7f16fb59e640 1 -- 172.21.15.59:0/2987895849 shutdown_connections 2024-03-29T11:26:28.573 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.587+0000 7f16fb59e640 1 -- 172.21.15.59:0/2987895849 >> 172.21.15.59:0/2987895849 conn(0x7f16f40058f0 msgr2=0x7f16f40043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:28.573 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.587+0000 7f16fb59e640 1 -- 172.21.15.59:0/2987895849 shutdown_connections 2024-03-29T11:26:28.573 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.587+0000 7f16fb59e640 1 -- 172.21.15.59:0/2987895849 wait complete. 2024-03-29T11:26:28.573 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:28.587+0000 7f16fb59e640 1 librados: shutdown 2024-03-29T11:26:28.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:35.397 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:35.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.557+0000 7fd17759e640 10 monclient: get_monmap_and_config 2024-03-29T11:26:35.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.557+0000 7fd17759e640 10 monclient: build_initial_monmap 2024-03-29T11:26:35.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.557+0000 7fd17759e640 10 monclient: monmap: 2024-03-29T11:26:35.544 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:35.544 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:35.544 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:26:35.558168+0000 2024-03-29T11:26:35.544 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:26:35.558168+0000 2024-03-29T11:26:35.544 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:35.545 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:35.545 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:35.545 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:26:35.545 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:26:35.545 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:35.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.558+0000 7fd17759e640 1 Processor -- start 2024-03-29T11:26:35.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.559+0000 7fd17759e640 1 -- start start 2024-03-29T11:26:35.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.559+0000 7fd17759e640 10 monclient: init 2024-03-29T11:26:35.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.560+0000 7fd17759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:35.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.560+0000 7fd17759e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:35.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.560+0000 7fd17759e640 10 monclient(hunting): picked mon.noname-c con 0x7fd1780a9750 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:35.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.560+0000 7fd17759e640 10 monclient(hunting): picked mon.noname-b con 0x7fd1780aad20 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:35.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.561+0000 7fd16dd9b640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fd1780aad20 0x7fd178141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:49662/0 (socket says 172.21.15.59:49662) 2024-03-29T11:26:35.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd16dd9b640 1 -- 172.21.15.59:0/2340006561 learned_addr learned my addr 172.21.15.59:0/2340006561 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:35.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd17759e640 10 monclient(hunting): picked mon.noname-a con 0x7fd1780a9b30 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:35.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd17759e640 1 -- 172.21.15.59:0/2340006561 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1780af4d0 con 0x7fd1780a9b30 2024-03-29T11:26:35.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd17759e640 1 -- 172.21.15.59:0/2340006561 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1780af7b0 con 0x7fd1780a9750 2024-03-29T11:26:35.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd17759e640 1 -- 172.21.15.59:0/2340006561 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1780acab0 con 0x7fd1780aad20 2024-03-29T11:26:35.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd17759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:35.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd17759e640 10 monclient(hunting): authenticate will time out at 1245.675922s 2024-03-29T11:26:35.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd175d9b640 1 -- 172.21.15.59:0/2340006561 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736499612 0 0) 0x7fd1780acab0 con 0x7fd1780aad20 2024-03-29T11:26:35.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd175d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd175d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd175d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd175d9b640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:26:35.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.562+0000 7fd175d9b640 1 -- 172.21.15.59:0/2340006561 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd164003440 con 0x7fd1780aad20 2024-03-29T11:26:35.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.563+0000 7fd175d9b640 1 -- 172.21.15.59:0/2340006561 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3227890664 0 0) 0x7fd164003440 con 0x7fd1780aad20 2024-03-29T11:26:35.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.563+0000 7fd175d9b640 1 -- 172.21.15.59:0/2340006561 >> v1:172.21.15.172:6789/0 conn(0x7fd1780a9750 legacy=0x7fd1780ac3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:35.561 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 1 -- 172.21.15.59:0/2340006561 >> v1:172.21.15.59:6789/0 conn(0x7fd1780a9b30 legacy=0x7fd178149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.561 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:26:35.561 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 1 monclient: found mon.noname-b 2024-03-29T11:26:35.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:35.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:26:35.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 1 -- 172.21.15.59:0/2340006561 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1780aba30 con 0x7fd1780aad20 2024-03-29T11:26:35.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 10 monclient: _finish_auth 0 2024-03-29T11:26:35.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:26:35.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:35.563 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 1 -- 172.21.15.59:0/2340006561 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd160001690 con 0x7fd1780aad20 2024-03-29T11:26:35.563 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.563 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:35.563 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.565+0000 7fd175d9b640 10 monclient: dump: 2024-03-29T11:26:35.563 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:35.564 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:35.564 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:35.564 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:35.564 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.564 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:35.564 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:35.564 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:35.565 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:35.565 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:35.565 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd17759e640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:26:35.566 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd175d9b640 1 -- 172.21.15.59:0/2340006561 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1600018b0 con 0x7fd1780aad20 2024-03-29T11:26:35.566 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd17759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:35.566 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd175d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:35.566 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd175d9b640 1 -- 172.21.15.59:0/2340006561 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd160001c40 con 0x7fd1780aad20 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd175d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd175d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd175d9b640 10 monclient: dump: 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:35.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd17759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:35.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd17759e640 4 set_mon_vals no callback set 2024-03-29T11:26:35.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd17759e640 10 monclient: shutdown 2024-03-29T11:26:35.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd17759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:35.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd17759e640 1 -- 172.21.15.59:0/2340006561 >> v1:172.21.15.59:6790/0 conn(0x7fd1780aad20 legacy=0x7fd178141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.566+0000 7fd17759e640 1 -- 172.21.15.59:0/2340006561 shutdown_connections 2024-03-29T11:26:35.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.567+0000 7fd17759e640 1 -- 172.21.15.59:0/2340006561 >> 172.21.15.59:0/2340006561 conn(0x7fd1780673e0 msgr2=0x7fd1780b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:35.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.567+0000 7fd17759e640 1 -- 172.21.15.59:0/2340006561 shutdown_connections 2024-03-29T11:26:35.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.568+0000 7fd17759e640 1 -- 172.21.15.59:0/2340006561 wait complete. 2024-03-29T11:26:35.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.569+0000 7fd17759e640 10 monclient: build_initial_monmap 2024-03-29T11:26:35.569 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.570+0000 7fd17759e640 10 monclient: monmap: 2024-03-29T11:26:35.569 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:35.569 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:35.569 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:26:35.569 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:26:35.569 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:35.569 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.570+0000 7fd17759e640 1 librados: starting msgr at 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.570+0000 7fd17759e640 1 librados: starting objecter 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.571+0000 7fd17759e640 1 Processor -- start 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.571+0000 7fd17759e640 1 -- start start 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.571+0000 7fd17759e640 1 librados: setting wanted keys 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.571+0000 7fd17759e640 1 librados: calling monclient init 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.571+0000 7fd17759e640 10 monclient: init 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.572+0000 7fd17759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:35.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.572+0000 7fd17759e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:35.571 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.572+0000 7fd17759e640 10 monclient(hunting): picked mon.noname-c con 0x7fd1780a9750 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:35.571 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.572+0000 7fd17759e640 10 monclient(hunting): picked mon.noname-a con 0x7fd1780a9b30 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:35.578 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.574+0000 7fd17659c640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fd1780a9750 0x7fd17806f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:52890/0 (socket says 172.21.15.59:52890) 2024-03-29T11:26:35.579 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.576+0000 7fd17659c640 1 -- 172.21.15.59:0/2520414409 learned_addr learned my addr 172.21.15.59:0/2520414409 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:35.579 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.576+0000 7fd17759e640 10 monclient(hunting): picked mon.noname-b con 0x7fd1780aad20 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:35.579 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.576+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd178070710 con 0x7fd1780a9b30 2024-03-29T11:26:35.579 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.576+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1780717b0 con 0x7fd1780aad20 2024-03-29T11:26:35.579 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.576+0000 7fd16f7fe640 10 client.?.objecter ms_handle_connect 0x7fd1780a9750 2024-03-29T11:26:35.579 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.576+0000 7fd16f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.576+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd178071a90 con 0x7fd1780a9750 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.576+0000 7fd17759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.576+0000 7fd17759e640 10 monclient(hunting): authenticate will time out at 1245.690612s 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.577+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237865037 0 0) 0x7fd178071a90 con 0x7fd1780a9750 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.577+0000 7fd16f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.577+0000 7fd16f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.577+0000 7fd16f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.577+0000 7fd16f7fe640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.577+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd158004eb0 con 0x7fd1780a9750 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 875324631 0 0) 0x7fd158004eb0 con 0x7fd1780a9750 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd178071a90 con 0x7fd1780a9750 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd168001870 con 0x7fd1780a9750 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:35.580 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient(hunting): dump: 2024-03-29T11:26:35.581 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:35.581 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:35.581 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:35.581 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:35.581 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.581 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:35.581 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:35.581 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:35.581 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:35.582 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:35.582 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:35.582 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2713226814 0 0) 0x7fd178071a90 con 0x7fd1780a9750 2024-03-29T11:26:35.582 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 >> v1:172.21.15.172:6789/0 conn(0x7fd1780aad20 legacy=0x7fd1781de150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.582 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 >> v1:172.21.15.59:6789/0 conn(0x7fd1780a9b30 legacy=0x7fd17813aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.582 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:35.582 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 1 monclient: found mon.c 2024-03-29T11:26:35.582 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:35.582 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:35.582 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd178070ba0 con 0x7fd1780a9750 2024-03-29T11:26:35.582 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1680019d0 con 0x7fd1780a9750 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd168004ae0 con 0x7fd1780a9750 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.578+0000 7fd16f7fe640 10 monclient: dump: 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.583 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.579+0000 7fd17559a640 4 set_mon_vals no callback set 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.580+0000 7fd17759e640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.580+0000 7fd17759e640 10 monclient: _renew_subs 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.580+0000 7fd17759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.580+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1780702e0 con 0x7fd1780a9750 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.580+0000 7fd17759e640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.580+0000 7fd17759e640 10 monclient: _renew_subs 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.580+0000 7fd17759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.580+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17806ff90 con 0x7fd1780a9750 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.580+0000 7fd17759e640 1 librados: init done 2024-03-29T11:26:35.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.582+0000 7fd17759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.582+0000 7fd17759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.582+0000 7fd17759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:35.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.582+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17804f730 con 0x7fd1780a9750 2024-03-29T11:26:35.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.586+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fd16801aab0 con 0x7fd1780a9750 2024-03-29T11:26:35.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.587+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fd168084aa0 con 0x7fd1780a9750 2024-03-29T11:26:35.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.587+0000 7fd16dd9b640 10 monclient: get_auth_request con 0x7fd158076540 auth_method 0 2024-03-29T11:26:35.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.587+0000 7fd16f7fe640 10 client.4490.objecter ms_dispatch 0x7fd178000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:35.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.587+0000 7fd16f7fe640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:35.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.587+0000 7fd16f7fe640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:35.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.587+0000 7fd16f7fe640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-29T11:26:35.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.588+0000 7fd16f7fe640 10 client.4490.objecter ms_handle_connect 0x7fd158076540 2024-03-29T11:26:35.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.593+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1680501b0 con 0x7fd1780a9750 2024-03-29T11:26:35.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.593+0000 7fd16f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.593+0000 7fd16f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:35.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.927+0000 7fd17759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:35.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.927+0000 7fd17759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:35.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.927+0000 7fd17759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:35.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.927+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd178074e10 con 0x7fd1780a9750 2024-03-29T11:26:35.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.929+0000 7fd16f7fe640 1 -- 172.21.15.59:0/2520414409 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fd1680522b0 con 0x7fd1780a9750 2024-03-29T11:26:35.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.929+0000 7fd16f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:35.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.929+0000 7fd16f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:35.917 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:26:35.917 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.933+0000 7fd17759e640 10 librados: watch_flush enter 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.933+0000 7fd17759e640 10 librados: watch_flush exit 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.933+0000 7fd17759e640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.933+0000 7fd17759e640 10 client.4490.objecter successfully canceled tick 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.934+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 >> v1:172.21.15.172:6816/2420742773 conn(0x7fd158076540 legacy=0x7fd158078a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.934+0000 7fd17759e640 10 monclient: shutdown 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.934+0000 7fd17759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:35.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.934+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 >> v1:172.21.15.59:6790/0 conn(0x7fd1780a9750 legacy=0x7fd17806f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.937+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 shutdown_connections 2024-03-29T11:26:35.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.937+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 >> 172.21.15.59:0/2520414409 conn(0x7fd1780673e0 msgr2=0x7fd1781417f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:35.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.937+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 shutdown_connections 2024-03-29T11:26:35.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.937+0000 7fd17759e640 1 -- 172.21.15.59:0/2520414409 wait complete. 2024-03-29T11:26:35.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:35.937+0000 7fd17759e640 1 librados: shutdown 2024-03-29T11:26:35.941 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:42.749 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.909+0000 7f67f0723640 10 monclient: get_monmap_and_config 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.909+0000 7f67f0723640 10 monclient: build_initial_monmap 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.909+0000 7f67f0723640 10 monclient: monmap: 2024-03-29T11:26:42.897 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:26:42.910524+0000 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:26:42.910524+0000 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:26:42.898 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:42.899 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.912+0000 7f67f0723640 1 Processor -- start 2024-03-29T11:26:42.899 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.912+0000 7f67f0723640 1 -- start start 2024-03-29T11:26:42.899 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.912+0000 7f67f0723640 10 monclient: init 2024-03-29T11:26:42.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.913+0000 7f67f0723640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:42.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.913+0000 7f67f0723640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:42.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.913+0000 7f67f0723640 10 monclient(hunting): picked mon.noname-a con 0x7f67e81457f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:42.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.913+0000 7f67f0723640 10 monclient(hunting): picked mon.noname-c con 0x7f67e813f650 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:42.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.913+0000 7f67f0723640 10 monclient(hunting): picked mon.noname-b con 0x7f67e80a9d70 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:42.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.913+0000 7f67f0723640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e80aaef0 con 0x7f67e81457f0 2024-03-29T11:26:42.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.913+0000 7f67f0723640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e8145bd0 con 0x7f67e813f650 2024-03-29T11:26:42.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.913+0000 7f67f0723640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e8145eb0 con 0x7f67e80a9d70 2024-03-29T11:26:42.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.913+0000 7f67f0723640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:42.902 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.913+0000 7f67f0723640 10 monclient(hunting): authenticate will time out at 1253.027667s 2024-03-29T11:26:42.902 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.914+0000 7f67edcbe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f67e813f650 0x7f67e80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:35330/0 (socket says 172.21.15.59:35330) 2024-03-29T11:26:42.902 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.914+0000 7f67edcbe640 1 -- 172.21.15.59:0/4270303072 learned_addr learned my addr 172.21.15.59:0/4270303072 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:42.906 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.919+0000 7f67ed4bd640 1 -- 172.21.15.59:0/4270303072 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407273984 0 0) 0x7f67e80aaef0 con 0x7f67e81457f0 2024-03-29T11:26:42.906 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.919+0000 7f67ed4bd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.919+0000 7f67ed4bd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.919+0000 7f67ed4bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.919+0000 7f67ed4bd640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.919+0000 7f67ed4bd640 1 -- 172.21.15.59:0/4270303072 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d4003520 con 0x7f67e81457f0 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 1 -- 172.21.15.59:0/4270303072 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616299510 0 0) 0x7f67e8145bd0 con 0x7f67e813f650 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 1 -- 172.21.15.59:0/4270303072 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e80aaef0 con 0x7f67e813f650 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 1 -- 172.21.15.59:0/4270303072 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 385961683 0 0) 0x7f67d4003520 con 0x7f67e81457f0 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 1 -- 172.21.15.59:0/4270303072 >> v1:172.21.15.59:6790/0 conn(0x7f67e80a9d70 legacy=0x7f67e814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 1 -- 172.21.15.59:0/4270303072 >> v1:172.21.15.172:6789/0 conn(0x7f67e813f650 legacy=0x7f67e80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 1 monclient: found mon.noname-a 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 1 -- 172.21.15.59:0/4270303072 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e8146340 con 0x7f67e81457f0 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient: _finish_auth 0 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 1 -- 172.21.15.59:0/4270303072 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f67e0001a10 con 0x7f67e81457f0 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient: dump: 2024-03-29T11:26:42.908 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:42.909 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:42.909 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:42.909 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:42.909 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.909 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:42.909 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:42.909 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:42.909 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:42.909 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:42.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67f0723640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:26:42.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67f0723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:42.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 1 -- 172.21.15.59:0/4270303072 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67e0001b70 con 0x7f67e81457f0 2024-03-29T11:26:42.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:42.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 1 -- 172.21.15.59:0/4270303072 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f67e0001f80 con 0x7f67e81457f0 2024-03-29T11:26:42.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:42.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.920+0000 7f67ed4bd640 10 monclient: dump: 2024-03-29T11:26:42.914 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:42.914 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:42.914 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:42.914 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:42.914 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.914 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:42.914 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.921+0000 7f67f0723640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.921+0000 7f67f0723640 4 set_mon_vals no callback set 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.921+0000 7f67f0723640 10 monclient: shutdown 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.921+0000 7f67f0723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.921+0000 7f67f0723640 1 -- 172.21.15.59:0/4270303072 >> v1:172.21.15.59:6789/0 conn(0x7f67e81457f0 legacy=0x7f67e813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.922+0000 7f67f0723640 1 -- 172.21.15.59:0/4270303072 shutdown_connections 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.922+0000 7f67f0723640 1 -- 172.21.15.59:0/4270303072 >> 172.21.15.59:0/4270303072 conn(0x7f67e8066a00 msgr2=0x7f67e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.923+0000 7f67f0723640 1 -- 172.21.15.59:0/4270303072 shutdown_connections 2024-03-29T11:26:42.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.923+0000 7f67f0723640 1 -- 172.21.15.59:0/4270303072 wait complete. 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.924+0000 7f67f0723640 10 monclient: build_initial_monmap 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.924+0000 7f67f0723640 10 monclient: monmap: 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:26:42.916 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:42.922 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.925+0000 7f67f0723640 1 librados: starting msgr at 2024-03-29T11:26:42.922 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.925+0000 7f67f0723640 1 librados: starting objecter 2024-03-29T11:26:42.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.925+0000 7f67f0723640 1 Processor -- start 2024-03-29T11:26:42.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.925+0000 7f67f0723640 1 -- start start 2024-03-29T11:26:42.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.925+0000 7f67f0723640 1 librados: setting wanted keys 2024-03-29T11:26:42.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.925+0000 7f67f0723640 1 librados: calling monclient init 2024-03-29T11:26:42.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.925+0000 7f67f0723640 10 monclient: init 2024-03-29T11:26:42.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.926+0000 7f67f0723640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:42.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.926+0000 7f67f0723640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:42.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.926+0000 7f67f0723640 10 monclient(hunting): picked mon.noname-a con 0x7f67e80a9d70 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:42.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.926+0000 7f67f0723640 10 monclient(hunting): picked mon.noname-b con 0x7f67e813f650 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:42.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.926+0000 7f67f0723640 10 monclient(hunting): picked mon.noname-c con 0x7f67e81457f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:42.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.927+0000 7f67f0723640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e81cbaa0 con 0x7f67e80a9d70 2024-03-29T11:26:42.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.927+0000 7f67f0723640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e81387a0 con 0x7f67e813f650 2024-03-29T11:26:42.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.927+0000 7f67f0723640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e81cbf30 con 0x7f67e81457f0 2024-03-29T11:26:42.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.927+0000 7f67f0723640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:42.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.927+0000 7f67f0723640 10 monclient(hunting): authenticate will time out at 1253.040928s 2024-03-29T11:26:42.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.927+0000 7f67edcbe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f67e813f650 0x7f67e8137980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:36132/0 (socket says 172.21.15.59:36132) 2024-03-29T11:26:42.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.927+0000 7f67edcbe640 1 -- 172.21.15.59:0/1120420201 learned_addr learned my addr 172.21.15.59:0/1120420201 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:42.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.927+0000 7f67deffd640 10 client.?.objecter ms_handle_connect 0x7f67e813f650 2024-03-29T11:26:42.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.927+0000 7f67deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:42.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620954546 0 0) 0x7f67e81387a0 con 0x7f67e813f650 2024-03-29T11:26:42.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:42.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:26:42.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67cc004eb0 con 0x7f67e813f650 2024-03-29T11:26:42.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3806474264 0 0) 0x7f67cc004eb0 con 0x7f67e813f650 2024-03-29T11:26:42.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67e81387a0 con 0x7f67e813f650 2024-03-29T11:26:42.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f67d80019b0 con 0x7f67e813f650 2024-03-29T11:26:42.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:42.926 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 10 monclient(hunting): dump: 2024-03-29T11:26:42.926 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:42.926 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:42.926 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:42.926 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:42.926 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.926 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:42.926 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:42.929 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:42.929 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.928+0000 7f67deffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.931+0000 7f67deffd640 10 client.?.objecter ms_handle_connect 0x7f67e81457f0 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.931+0000 7f67deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.931+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1477897449 0 0) 0x7f67e81387a0 con 0x7f67e813f650 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.931+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 >> v1:172.21.15.59:6790/0 conn(0x7f67e81457f0 legacy=0x7f67e8138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.931+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 >> v1:172.21.15.59:6789/0 conn(0x7f67e80a9d70 legacy=0x7f67e81cd4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.931+0000 7f67deffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.931+0000 7f67deffd640 1 monclient: found mon.b 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.931+0000 7f67deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.931+0000 7f67deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.931+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e81ccdd0 con 0x7f67e813f650 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.931+0000 7f67deffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:42.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.932+0000 7f67deffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.932+0000 7f67deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.932+0000 7f67f0723640 5 monclient: authenticate success, global_id 4480 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.932+0000 7f67f0723640 10 monclient: _renew_subs 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.932+0000 7f67f0723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.932+0000 7f67f0723640 1 -- 172.21.15.59:0/1120420201 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e81cd0b0 con 0x7f67e813f650 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.932+0000 7f67f0723640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.932+0000 7f67f0723640 10 monclient: _renew_subs 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.932+0000 7f67f0723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.932+0000 7f67f0723640 1 -- 172.21.15.59:0/1120420201 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e8142ba0 con 0x7f67e813f650 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.932+0000 7f67f0723640 1 librados: init done 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.935+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67d80047e0 con 0x7f67e813f650 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.935+0000 7f67deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.935+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f67d8004bf0 con 0x7f67e813f650 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.935+0000 7f67deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.935+0000 7f67deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.935+0000 7f67deffd640 10 monclient: dump: 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.936+0000 7f67dffff640 4 set_mon_vals no callback set 2024-03-29T11:26:42.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.936+0000 7f67f0723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:42.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.936+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f67d8004e10 con 0x7f67e813f650 2024-03-29T11:26:42.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.937+0000 7f67f0723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:42.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.937+0000 7f67f0723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:26:42.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.937+0000 7f67f0723640 1 -- 172.21.15.59:0/1120420201 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d0005080 con 0x7f67e813f650 2024-03-29T11:26:42.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.937+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f67d8084bd0 con 0x7f67e813f650 2024-03-29T11:26:42.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.937+0000 7f67deffd640 10 client.4480.objecter ms_dispatch 0x7f67e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:42.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.937+0000 7f67deffd640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:42.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.937+0000 7f67deffd640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:42.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.937+0000 7f67deffd640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-29T11:26:42.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.938+0000 7f67ee4bf640 10 monclient: get_auth_request con 0x7f67cc06dc70 auth_method 0 2024-03-29T11:26:42.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.939+0000 7f67deffd640 10 client.4480.objecter ms_handle_connect 0x7f67cc06dc70 2024-03-29T11:26:42.935 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.947+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67d80504b0 con 0x7f67e813f650 2024-03-29T11:26:42.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.947+0000 7f67deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:42.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:42.947+0000 7f67deffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:43.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.272+0000 7f67f0723640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.272+0000 7f67f0723640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.272+0000 7f67f0723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:26:43.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.272+0000 7f67f0723640 1 -- 172.21.15.59:0/1120420201 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67d0005330 con 0x7f67e813f650 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.274+0000 7f67deffd640 1 -- 172.21.15.59:0/1120420201 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f67d80523e0 con 0x7f67e813f650 2024-03-29T11:26:43.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.274+0000 7f67deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.274+0000 7f67deffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:43.261 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:26:43.262 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:43.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.278+0000 7f67f0723640 10 librados: watch_flush enter 2024-03-29T11:26:43.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.278+0000 7f67f0723640 10 librados: watch_flush exit 2024-03-29T11:26:43.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.278+0000 7f67f0723640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-29T11:26:43.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.278+0000 7f67f0723640 10 client.4480.objecter successfully canceled tick 2024-03-29T11:26:43.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.278+0000 7f67f0723640 1 -- 172.21.15.59:0/1120420201 >> v1:172.21.15.172:6816/2420742773 conn(0x7f67cc06dc70 legacy=0x7f67cc070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.278+0000 7f67f0723640 10 monclient: shutdown 2024-03-29T11:26:43.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.278+0000 7f67f0723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:43.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.278+0000 7f67f0723640 1 -- 172.21.15.59:0/1120420201 >> v1:172.21.15.172:6789/0 conn(0x7f67e813f650 legacy=0x7f67e8137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.278+0000 7f67f0723640 1 -- 172.21.15.59:0/1120420201 shutdown_connections 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.278+0000 7f67f0723640 1 -- 172.21.15.59:0/1120420201 >> 172.21.15.59:0/1120420201 conn(0x7f67e8066a00 msgr2=0x7f67e80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.279+0000 7f67f0723640 1 -- 172.21.15.59:0/1120420201 shutdown_connections 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.279+0000 7f67f0723640 1 -- 172.21.15.59:0/1120420201 wait complete. 2024-03-29T11:26:43.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:43.279+0000 7f67f0723640 1 librados: shutdown 2024-03-29T11:26:43.284 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:50.089 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:50.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.264+0000 7f1aa332d640 10 monclient: get_monmap_and_config 2024-03-29T11:26:50.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.264+0000 7f1aa332d640 10 monclient: build_initial_monmap 2024-03-29T11:26:50.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.264+0000 7f1aa332d640 10 monclient: monmap: 2024-03-29T11:26:50.254 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:50.254 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:50.254 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:26:50.265328+0000 2024-03-29T11:26:50.254 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:26:50.265328+0000 2024-03-29T11:26:50.255 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:50.255 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:50.255 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:50.255 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:26:50.255 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:26:50.255 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:50.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.265+0000 7f1aa332d640 1 Processor -- start 2024-03-29T11:26:50.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.265+0000 7f1aa332d640 1 -- start start 2024-03-29T11:26:50.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.265+0000 7f1aa332d640 10 monclient: init 2024-03-29T11:26:50.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.266+0000 7f1aa332d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:50.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.266+0000 7f1aa332d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:50.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.266+0000 7f1aa332d640 10 monclient(hunting): picked mon.noname-c con 0x7f1a9c0aad20 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:50.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.270+0000 7f1aa332d640 10 monclient(hunting): picked mon.noname-b con 0x7f1a9c0ad2f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:50.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.270+0000 7f1aa332d640 10 monclient(hunting): picked mon.noname-a con 0x7f1a9c0ac3a0 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:50.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.270+0000 7f1aa332d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c0ab750 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.270+0000 7f1aa332d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c1535d0 con 0x7f1a9c0aad20 2024-03-29T11:26:50.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.270+0000 7f1aa332d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c0ab9c0 con 0x7f1a9c0ad2f0 2024-03-29T11:26:50.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.270+0000 7f1aa332d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:50.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.270+0000 7f1aa332d640 10 monclient(hunting): authenticate will time out at 1260.384195s 2024-03-29T11:26:50.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.273+0000 7f1aa232b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1a9c0aad20 0x7f1a9c1527b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:43495/0 (socket says 172.21.15.59:43495) 2024-03-29T11:26:50.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.273+0000 7f1aa232b640 1 -- 172.21.15.59:0/504206653 learned_addr learned my addr 172.21.15.59:0/504206653 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:50.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.274+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939232910 0 0) 0x7f1a9c1535d0 con 0x7f1a9c0aad20 2024-03-29T11:26:50.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.274+0000 7f1aa1329640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.274+0000 7f1aa1329640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.274+0000 7f1aa1329640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.274+0000 7f1aa1329640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:26:50.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.275+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a88003520 con 0x7f1a9c0aad20 2024-03-29T11:26:50.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.275+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012807589 0 0) 0x7f1a9c0ab9c0 con 0x7f1a9c0ad2f0 2024-03-29T11:26:50.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.275+0000 7f1aa1329640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.275+0000 7f1aa1329640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.275+0000 7f1aa1329640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.275+0000 7f1aa1329640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:26:50.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.275+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a9c1535d0 con 0x7f1a9c0ad2f0 2024-03-29T11:26:50.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.276+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074503872 0 0) 0x7f1a9c0ab750 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.276+0000 7f1aa1329640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.276+0000 7f1aa1329640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.276+0000 7f1aa1329640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.276+0000 7f1aa1329640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:26:50.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.276+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a9c0ab9c0 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.276+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 186590805 0 0) 0x7f1a88003520 con 0x7f1a9c0aad20 2024-03-29T11:26:50.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.276+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 >> v1:172.21.15.59:6790/0 conn(0x7f1a9c0ad2f0 legacy=0x7f1a9c152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.266 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 >> v1:172.21.15.59:6789/0 conn(0x7f1a9c0ac3a0 legacy=0x7f1a9c15aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 10 monclient: _finish_hunting 0 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 1 monclient: found mon.noname-c 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c0abca0 con 0x7f1a9c0aad20 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 10 monclient: _finish_auth 0 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 10 monclient: _check_auth_tickets 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1a98001690 con 0x7f1a9c0aad20 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 10 monclient: dump: 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:50.267 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a980018b0 con 0x7f1a9c0aad20 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 1 -- 172.21.15.59:0/504206653 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1a98001c40 con 0x7f1a9c0aad20 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.277+0000 7f1aa1329640 10 monclient: dump: 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:50.268 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:50.269 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:50.269 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:50.269 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.269 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:50.269 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:50.269 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:50.269 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:50.269 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:50.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.278+0000 7f1aa332d640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:26:50.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.278+0000 7f1aa332d640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:50.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.278+0000 7f1aa332d640 4 set_mon_vals no callback set 2024-03-29T11:26:50.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.279+0000 7f1aa332d640 10 monclient: shutdown 2024-03-29T11:26:50.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.279+0000 7f1aa332d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:50.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.279+0000 7f1aa332d640 1 -- 172.21.15.59:0/504206653 >> v1:172.21.15.172:6789/0 conn(0x7f1a9c0aad20 legacy=0x7f1a9c1527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.279+0000 7f1aa332d640 1 -- 172.21.15.59:0/504206653 shutdown_connections 2024-03-29T11:26:50.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.279+0000 7f1aa332d640 1 -- 172.21.15.59:0/504206653 >> 172.21.15.59:0/504206653 conn(0x7f1a9c0673e0 msgr2=0x7f1a9c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:50.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.283+0000 7f1aa332d640 1 -- 172.21.15.59:0/504206653 shutdown_connections 2024-03-29T11:26:50.271 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.283+0000 7f1aa332d640 1 -- 172.21.15.59:0/504206653 wait complete. 2024-03-29T11:26:50.271 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.284+0000 7f1aa332d640 10 monclient: build_initial_monmap 2024-03-29T11:26:50.271 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.284+0000 7f1aa332d640 10 monclient: monmap: 2024-03-29T11:26:50.272 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:50.272 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:50.272 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:26:50.272 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:26:50.272 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:50.272 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:50.272 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:50.272 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:26:50.272 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:26:50.272 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:50.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.284+0000 7f1aa332d640 1 librados: starting msgr at 2024-03-29T11:26:50.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.284+0000 7f1aa332d640 1 librados: starting objecter 2024-03-29T11:26:50.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.285+0000 7f1aa332d640 1 Processor -- start 2024-03-29T11:26:50.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.285+0000 7f1aa332d640 1 -- start start 2024-03-29T11:26:50.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.285+0000 7f1aa332d640 1 librados: setting wanted keys 2024-03-29T11:26:50.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.285+0000 7f1aa332d640 1 librados: calling monclient init 2024-03-29T11:26:50.273 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.285+0000 7f1aa332d640 10 monclient: init 2024-03-29T11:26:50.274 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.285+0000 7f1aa332d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:50.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.285+0000 7f1aa332d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:50.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.285+0000 7f1aa332d640 10 monclient(hunting): picked mon.noname-b con 0x7f1a9c0aad20 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:50.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.287+0000 7f1aa332d640 10 monclient(hunting): picked mon.noname-a con 0x7f1a9c0ac3a0 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:50.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.287+0000 7f1aa332d640 10 monclient(hunting): picked mon.noname-c con 0x7f1a9c0ad2f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:50.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.287+0000 7f1aa332d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c148b80 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.287+0000 7f1aa332d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c14c4f0 con 0x7f1a9c0aad20 2024-03-29T11:26:50.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.287+0000 7f1aa332d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c08b180 con 0x7f1a9c0ad2f0 2024-03-29T11:26:50.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.287+0000 7f1aa332d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:50.275 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.287+0000 7f1aa332d640 10 monclient(hunting): authenticate will time out at 1260.401424s 2024-03-29T11:26:50.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.289+0000 7f1aa1b2a640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1a9c0ac3a0 0x7f1a9c14bde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38103/0 (socket says 172.21.15.59:38103) 2024-03-29T11:26:50.276 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.289+0000 7f1aa1b2a640 1 -- 172.21.15.59:0/1963808961 learned_addr learned my addr 172.21.15.59:0/1963808961 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:50.281 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.292+0000 7f1a92ffd640 10 client.?.objecter ms_handle_connect 0x7f1a9c0ac3a0 2024-03-29T11:26:50.281 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.293+0000 7f1a92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:50.281 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.293+0000 7f1a92ffd640 10 client.?.objecter ms_handle_connect 0x7f1a9c0ad2f0 2024-03-29T11:26:50.282 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.293+0000 7f1a92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:50.282 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.293+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970175489 0 0) 0x7f1a9c148b80 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.282 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.293+0000 7f1a92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.282 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.293+0000 7f1a92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.282 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.293+0000 7f1a92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.282 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.293+0000 7f1a92ffd640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:26:50.282 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.293+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a80004eb0 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.282 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.293+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498342474 0 0) 0x7f1a9c08b180 con 0x7f1a9c0ad2f0 2024-03-29T11:26:50.282 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.294+0000 7f1a92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.283 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.294+0000 7f1a92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.283 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.294+0000 7f1a92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.283 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.294+0000 7f1a92ffd640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:26:50.283 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.294+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a9c148b80 con 0x7f1a9c0ad2f0 2024-03-29T11:26:50.283 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.295+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2984871104 0 0) 0x7f1a80004eb0 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.283 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.295+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a9c08b180 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.289 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.295+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1a94001d10 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.289 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.295+0000 7f1a92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.290 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.295+0000 7f1a92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:50.290 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.295+0000 7f1a92ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:50.290 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:50.290 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:50.290 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:50.290 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:50.290 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.290 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:50.291 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:50.291 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:50.291 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:50.291 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:50.291 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.295+0000 7f1a92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:50.291 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.295+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 944263786 0 0) 0x7f1a9c08b180 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.291 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.295+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 >> v1:172.21.15.59:6790/0 conn(0x7f1a9c0ad2f0 legacy=0x7f1a9c08a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.291 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 >> v1:172.21.15.172:6789/0 conn(0x7f1a9c0aad20 legacy=0x7f1a9c1e72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1a92ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:50.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1a92ffd640 1 monclient: found mon.a 2024-03-29T11:26:50.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1a92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:50.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1a92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:26:50.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c08b310 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1a92ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:50.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1a92ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:50.292 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1a92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:50.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a94001e70 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1a92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:50.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1aa0b28640 4 set_mon_vals no callback set 2024-03-29T11:26:50.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1aa332d640 5 monclient: authenticate success, global_id 4491 2024-03-29T11:26:50.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1aa332d640 10 monclient: _renew_subs 2024-03-29T11:26:50.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1aa332d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:26:50.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1aa332d640 1 -- 172.21.15.59:0/1963808961 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9c08acd0 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1aa332d640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:50.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1aa332d640 10 monclient: _renew_subs 2024-03-29T11:26:50.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1aa332d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:26:50.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1aa332d640 1 -- 172.21.15.59:0/1963808961 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9c1e7ea0 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.296+0000 7f1aa332d640 1 librados: init done 2024-03-29T11:26:50.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.297+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1a94003c40 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.297+0000 7f1a92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.297+0000 7f1a92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:50.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.297+0000 7f1a92ffd640 10 monclient: dump: 2024-03-29T11:26:50.294 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:50.294 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.299+0000 7f1aa332d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.299+0000 7f1aa332d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.299+0000 7f1aa332d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.299+0000 7f1aa332d640 1 -- 172.21.15.59:0/1963808961 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9c04f730 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.302+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f1a94003da0 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.302+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f1a94084db0 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.302+0000 7f1a92ffd640 10 client.4491.objecter ms_dispatch 0x7f1a9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:50.296 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.302+0000 7f1a92ffd640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:50.296 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.302+0000 7f1a92ffd640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:50.296 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.302+0000 7f1a92ffd640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-29T11:26:50.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.306+0000 7f1aa232b640 10 monclient: get_auth_request con 0x7f1a8006dcb0 auth_method 0 2024-03-29T11:26:50.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.307+0000 7f1a92ffd640 10 client.4491.objecter ms_handle_connect 0x7f1a8006dcb0 2024-03-29T11:26:50.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.308+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a94050690 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.308+0000 7f1a92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.297 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.308+0000 7f1a92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:50.617 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.630+0000 7f1aa332d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.631+0000 7f1aa332d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.631+0000 7f1aa332d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:26:50.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.631+0000 7f1aa332d640 1 -- 172.21.15.59:0/1963808961 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a9c074e10 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.631+0000 7f1a92ffd640 1 -- 172.21.15.59:0/1963808961 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f1a940525c0 con 0x7f1a9c0ac3a0 2024-03-29T11:26:50.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.631+0000 7f1a92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.631+0000 7f1a92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:50.619 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:26:50.619 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.635+0000 7f1aa332d640 10 librados: watch_flush enter 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.635+0000 7f1aa332d640 10 librados: watch_flush exit 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.635+0000 7f1aa332d640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-29T11:26:50.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.635+0000 7f1aa332d640 10 client.4491.objecter successfully canceled tick 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.635+0000 7f1aa332d640 1 -- 172.21.15.59:0/1963808961 >> v1:172.21.15.172:6816/2420742773 conn(0x7f1a8006dcb0 legacy=0x7f1a80070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.635+0000 7f1aa332d640 10 monclient: shutdown 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.635+0000 7f1aa332d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.635+0000 7f1aa332d640 1 -- 172.21.15.59:0/1963808961 >> v1:172.21.15.59:6789/0 conn(0x7f1a9c0ac3a0 legacy=0x7f1a9c14bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.635+0000 7f1aa332d640 1 -- 172.21.15.59:0/1963808961 shutdown_connections 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.635+0000 7f1aa332d640 1 -- 172.21.15.59:0/1963808961 >> 172.21.15.59:0/1963808961 conn(0x7f1a9c0673e0 msgr2=0x7f1a9c069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:50.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.636+0000 7f1aa332d640 1 -- 172.21.15.59:0/1963808961 shutdown_connections 2024-03-29T11:26:50.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.636+0000 7f1aa332d640 1 -- 172.21.15.59:0/1963808961 wait complete. 2024-03-29T11:26:50.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:50.636+0000 7f1aa332d640 1 librados: shutdown 2024-03-29T11:26:50.640 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:57.447 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:57.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.612+0000 7f82c892a640 10 monclient: get_monmap_and_config 2024-03-29T11:26:57.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.612+0000 7f82c892a640 10 monclient: build_initial_monmap 2024-03-29T11:26:57.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.612+0000 7f82c892a640 10 monclient: monmap: 2024-03-29T11:26:57.599 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:57.599 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:57.599 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:26:57.612944+0000 2024-03-29T11:26:57.599 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:26:57.612944+0000 2024-03-29T11:26:57.600 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:57.600 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:57.600 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:57.600 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:26:57.600 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:26:57.600 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:57.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.613+0000 7f82c892a640 1 Processor -- start 2024-03-29T11:26:57.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.613+0000 7f82c892a640 1 -- start start 2024-03-29T11:26:57.600 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.613+0000 7f82c892a640 10 monclient: init 2024-03-29T11:26:57.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c892a640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:57.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c892a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:57.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c892a640 10 monclient(hunting): picked mon.noname-b con 0x7f82bc0389e0 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:57.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c892a640 10 monclient(hunting): picked mon.noname-a con 0x7f82bc038dc0 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:57.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c892a640 10 monclient(hunting): picked mon.noname-c con 0x7f82bc030660 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:57.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c892a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc0c54c0 con 0x7f82bc038dc0 2024-03-29T11:26:57.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c892a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc036430 con 0x7f82bc030660 2024-03-29T11:26:57.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c892a640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc039840 con 0x7f82bc0389e0 2024-03-29T11:26:57.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c892a640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:57.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c892a640 10 monclient(hunting): authenticate will time out at 1267.728248s 2024-03-29T11:26:57.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c259c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f82bc038dc0 0x7f82bc0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51158/0 (socket says 172.21.15.59:51158) 2024-03-29T11:26:57.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.614+0000 7f82c259c640 1 -- 172.21.15.59:0/3351018335 learned_addr learned my addr 172.21.15.59:0/3351018335 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:57.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939938459 0 0) 0x7f82bc036430 con 0x7f82bc030660 2024-03-29T11:26:57.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:26:57.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82ac003520 con 0x7f82bc030660 2024-03-29T11:26:57.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885863865 0 0) 0x7f82bc0c54c0 con 0x7f82bc038dc0 2024-03-29T11:26:57.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:26:57.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc036430 con 0x7f82bc038dc0 2024-03-29T11:26:57.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616409814 0 0) 0x7f82bc039840 con 0x7f82bc0389e0 2024-03-29T11:26:57.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:26:57.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc0c54c0 con 0x7f82bc0389e0 2024-03-29T11:26:57.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1919075147 0 0) 0x7f82ac003520 con 0x7f82bc030660 2024-03-29T11:26:57.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 >> v1:172.21.15.59:6790/0 conn(0x7f82bc0389e0 legacy=0x7f82bc033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 >> v1:172.21.15.59:6789/0 conn(0x7f82bc038dc0 legacy=0x7f82bc0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:26:57.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 monclient: found mon.noname-c 2024-03-29T11:26:57.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:57.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:26:57.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc035690 con 0x7f82bc030660 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c892a640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c892a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f82c404f730 con 0x7f82bc030660 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c1d9b640 10 monclient: dump: 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:57.607 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:57.608 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.608 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:57.608 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:57.608 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:57.608 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:57.608 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:57.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.615+0000 7f82c892a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:57.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.618+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82c4075f10 con 0x7f82bc030660 2024-03-29T11:26:57.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.618+0000 7f82c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:57.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.618+0000 7f82c892a640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:57.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.618+0000 7f82c892a640 4 set_mon_vals no callback set 2024-03-29T11:26:57.616 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.618+0000 7f82c1d9b640 1 -- 172.21.15.59:0/3351018335 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f82c40758b0 con 0x7f82bc030660 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.618+0000 7f82c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.618+0000 7f82c1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.618+0000 7f82c1d9b640 10 monclient: dump: 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:57.617 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.619+0000 7f82c892a640 10 monclient: shutdown 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.619+0000 7f82c892a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.619+0000 7f82c892a640 1 -- 172.21.15.59:0/3351018335 >> v1:172.21.15.172:6789/0 conn(0x7f82bc030660 legacy=0x7f82bc0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.619+0000 7f82c892a640 1 -- 172.21.15.59:0/3351018335 shutdown_connections 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.619+0000 7f82c892a640 1 -- 172.21.15.59:0/3351018335 >> 172.21.15.59:0/3351018335 conn(0x7f82bc005c00 msgr2=0x7f82bc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.619+0000 7f82c892a640 1 -- 172.21.15.59:0/3351018335 shutdown_connections 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.619+0000 7f82c892a640 1 -- 172.21.15.59:0/3351018335 wait complete. 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.620+0000 7f82c892a640 10 monclient: build_initial_monmap 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.620+0000 7f82c892a640 10 monclient: monmap: 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:26:57.618 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:57.619 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:57.619 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:26:57.619 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:26:57.619 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:26:57.619 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:57.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.620+0000 7f82c892a640 1 librados: starting msgr at 2024-03-29T11:26:57.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.620+0000 7f82c892a640 1 librados: starting objecter 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.621+0000 7f82c892a640 1 Processor -- start 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.621+0000 7f82c892a640 1 -- start start 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.621+0000 7f82c892a640 1 librados: setting wanted keys 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.621+0000 7f82c892a640 1 librados: calling monclient init 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.621+0000 7f82c892a640 10 monclient: init 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c892a640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c892a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c892a640 10 monclient(hunting): picked mon.noname-c con 0x7f82bc030660 addr v1:172.21.15.59:6790/0 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c892a640 10 monclient(hunting): picked mon.noname-b con 0x7f82bc0389e0 addr v1:172.21.15.172:6789/0 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c892a640 10 monclient(hunting): picked mon.noname-a con 0x7f82bc02f420 addr v1:172.21.15.59:6789/0 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c359e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f82bc02f420 0x7f82bc164410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51852/0 (socket says 172.21.15.59:51852) 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c359e640 1 -- 172.21.15.59:0/2418294983 learned_addr learned my addr 172.21.15.59:0/2418294983 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82ab7fe640 10 client.?.objecter ms_handle_connect 0x7f82bc02f420 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:57.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82ab7fe640 10 client.?.objecter ms_handle_connect 0x7f82bc030660 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc02f870 con 0x7f82bc02f420 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc030010 con 0x7f82bc0389e0 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc0c8840 con 0x7f82bc030660 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c892a640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.622+0000 7f82c892a640 10 monclient(hunting): authenticate will time out at 1267.736399s 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792885762 0 0) 0x7f82bc0c8840 con 0x7f82bc030660 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b0004eb0 con 0x7f82bc030660 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344892257 0 0) 0x7f82bc02f870 con 0x7f82bc02f420 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.621 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc0c8840 con 0x7f82bc02f420 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 client.?.objecter ms_handle_connect 0x7f82bc0389e0 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3300447310 0 0) 0x7f82b0004eb0 con 0x7f82bc030660 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82bc02f870 con 0x7f82bc030660 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f82b8001930 con 0x7f82bc030660 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): dump: 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:57.622 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2506993427 0 0) 0x7f82bc02f870 con 0x7f82bc030660 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 >> v1:172.21.15.172:6789/0 conn(0x7f82bc0389e0 legacy=0x7f82bc0c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.623+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 >> v1:172.21.15.59:6789/0 conn(0x7f82bc02f420 legacy=0x7f82bc164410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 1 monclient: found mon.c 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:57.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0bfbf0 con 0x7f82bc030660 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82c892a640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82c892a640 10 monclient: _renew_subs 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82c892a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc02faf0 con 0x7f82bc030660 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82c892a640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82c892a640 10 monclient: _renew_subs 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82c892a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc02eaf0 con 0x7f82bc030660 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82c892a640 1 librados: init done 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82b8001d60 con 0x7f82bc030660 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:57.624 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f82b8003a60 con 0x7f82bc030660 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.624+0000 7f82ab7fe640 10 monclient: dump: 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.625+0000 7f82c892a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.625 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.625+0000 7f82c159a640 4 set_mon_vals no callback set 2024-03-29T11:26:57.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.627+0000 7f82c892a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.627+0000 7f82c892a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:57.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.627+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82bc007b90 con 0x7f82bc030660 2024-03-29T11:26:57.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.628+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f82b8019a30 con 0x7f82bc030660 2024-03-29T11:26:57.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.629+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f82b80838f0 con 0x7f82bc030660 2024-03-29T11:26:57.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.629+0000 7f82ab7fe640 10 client.4505.objecter ms_dispatch 0x7f82c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:57.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.629+0000 7f82ab7fe640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:57.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.629+0000 7f82ab7fe640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:57.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.629+0000 7f82c259c640 10 monclient: get_auth_request con 0x7f82b006da00 auth_method 0 2024-03-29T11:26:57.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.629+0000 7f82ab7fe640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:26:57.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.630+0000 7f82ab7fe640 10 client.4505.objecter ms_handle_connect 0x7f82b006da00 2024-03-29T11:26:57.628 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.637+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82b804f1d0 con 0x7f82bc030660 2024-03-29T11:26:57.628 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.637+0000 7f82ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.628 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:57.637+0000 7f82ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:58.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.022+0000 7f82c892a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:58.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.022+0000 7f82c892a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:58.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.022+0000 7f82c892a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:26:58.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.022+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82bc005120 con 0x7f82bc030660 2024-03-29T11:26:58.010 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.023+0000 7f82ab7fe640 1 -- 172.21.15.59:0/2418294983 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f82b8051100 con 0x7f82bc030660 2024-03-29T11:26:58.010 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:26:58.011 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:58.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.023+0000 7f82ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:58.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.023+0000 7f82ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:58.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.027+0000 7f82c892a640 10 librados: watch_flush enter 2024-03-29T11:26:58.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.027+0000 7f82c892a640 10 librados: watch_flush exit 2024-03-29T11:26:58.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.027+0000 7f82c892a640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:26:58.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.027+0000 7f82c892a640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:26:58.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.027+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 >> v1:172.21.15.172:6816/2420742773 conn(0x7f82b006da00 legacy=0x7f82b006fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.027+0000 7f82c892a640 10 monclient: shutdown 2024-03-29T11:26:58.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.027+0000 7f82c892a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:58.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.027+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 >> v1:172.21.15.59:6790/0 conn(0x7f82bc030660 legacy=0x7f82bc0ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.027+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 shutdown_connections 2024-03-29T11:26:58.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.027+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 >> 172.21.15.59:0/2418294983 conn(0x7f82bc005c00 msgr2=0x7f82bc0c3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:58.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.028+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 shutdown_connections 2024-03-29T11:26:58.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.028+0000 7f82c892a640 1 -- 172.21.15.59:0/2418294983 wait complete. 2024-03-29T11:26:58.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:26:58.028+0000 7f82c892a640 1 librados: shutdown 2024-03-29T11:26:58.031 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:04.839 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:04.985 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.997+0000 7fbdeb1fd640 10 monclient: get_monmap_and_config 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.997+0000 7fbdeb1fd640 10 monclient: build_initial_monmap 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.997+0000 7fbdeb1fd640 10 monclient: monmap: 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:27:04.999124+0000 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:27:04.999124+0000 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:27:04.986 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:04.989 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.998+0000 7fbdeb1fd640 1 Processor -- start 2024-03-29T11:27:04.989 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.998+0000 7fbdeb1fd640 1 -- start start 2024-03-29T11:27:04.989 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.998+0000 7fbdeb1fd640 10 monclient: init 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.999+0000 7fbdeb1fd640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.999+0000 7fbdeb1fd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.999+0000 7fbdeb1fd640 10 monclient(hunting): picked mon.noname-b con 0x7fbddc006e30 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.999+0000 7fbdeb1fd640 10 monclient(hunting): picked mon.noname-a con 0x7fbddc035e00 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.999+0000 7fbdeb1fd640 10 monclient(hunting): picked mon.noname-c con 0x7fbddc0c9670 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.999+0000 7fbdeb1fd640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc035970 con 0x7fbddc035e00 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.999+0000 7fbdeb1fd640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc036280 con 0x7fbddc0c9670 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.999+0000 7fbdeb1fd640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc035640 con 0x7fbddc006e30 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.999+0000 7fbdeb1fd640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:04.999+0000 7fbdeb1fd640 10 monclient(hunting): authenticate will time out at 1275.114561s 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.000+0000 7fbdea1fb640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fbddc006e30 0x7fbddc0cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:58814/0 (socket says 172.21.15.59:58814) 2024-03-29T11:27:04.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.000+0000 7fbdea1fb640 1 -- 172.21.15.59:0/3729859328 learned_addr learned my addr 172.21.15.59:0/3729859328 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:04.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 1 -- 172.21.15.59:0/3729859328 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961615987 0 0) 0x7fbddc036280 con 0x7fbddc0c9670 2024-03-29T11:27:04.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:04.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:04.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:04.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:27:04.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 1 -- 172.21.15.59:0/3729859328 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd0003520 con 0x7fbddc0c9670 2024-03-29T11:27:04.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 1 -- 172.21.15.59:0/3729859328 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3754447001 0 0) 0x7fbdd0003520 con 0x7fbddc0c9670 2024-03-29T11:27:04.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 1 -- 172.21.15.59:0/3729859328 >> v1:172.21.15.59:6790/0 conn(0x7fbddc006e30 legacy=0x7fbddc0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:04.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 1 -- 172.21.15.59:0/3729859328 >> v1:172.21.15.59:6789/0 conn(0x7fbddc035e00 legacy=0x7fbddc0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:04.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:04.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 1 monclient: found mon.noname-c 2024-03-29T11:27:04.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:04.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:27:04.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 1 -- 172.21.15.59:0/3729859328 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc034e20 con 0x7fbddc0c9670 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 10 monclient: _finish_auth 0 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 1 -- 172.21.15.59:0/3729859328 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fbde404f730 con 0x7fbddc0c9670 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbde91f9640 10 monclient: dump: 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:04.995 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.005+0000 7fbdeb1fd640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.006+0000 7fbdeb1fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.006+0000 7fbde91f9640 1 -- 172.21.15.59:0/3729859328 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbde4075f10 con 0x7fbddc0c9670 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.006+0000 7fbde91f9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.006+0000 7fbdeb1fd640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.006+0000 7fbde91f9640 1 -- 172.21.15.59:0/3729859328 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fbde40758b0 con 0x7fbddc0c9670 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.006+0000 7fbde91f9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.006+0000 7fbdeb1fd640 4 set_mon_vals no callback set 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.006+0000 7fbde91f9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:04.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.006+0000 7fbde91f9640 10 monclient: dump: 2024-03-29T11:27:04.997 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:04.997 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:04.997 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:04.997 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:04.997 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:04.997 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:04.997 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:04.997 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:04.997 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:04.997 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:04.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.007+0000 7fbdeb1fd640 10 monclient: shutdown 2024-03-29T11:27:04.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.007+0000 7fbdeb1fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:04.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.007+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/3729859328 >> v1:172.21.15.172:6789/0 conn(0x7fbddc0c9670 legacy=0x7fbddc0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:04.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.007+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/3729859328 shutdown_connections 2024-03-29T11:27:04.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.007+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/3729859328 >> 172.21.15.59:0/3729859328 conn(0x7fbddc005c00 msgr2=0x7fbddc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:04.998 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.010+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/3729859328 shutdown_connections 2024-03-29T11:27:05.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.012+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/3729859328 wait complete. 2024-03-29T11:27:05.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.013+0000 7fbdeb1fd640 10 monclient: build_initial_monmap 2024-03-29T11:27:05.001 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.013+0000 7fbdeb1fd640 10 monclient: monmap: 2024-03-29T11:27:05.001 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:05.002 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:05.002 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:27:05.003 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:27:05.003 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:05.003 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:05.003 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:05.003 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:27:05.003 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:27:05.003 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:05.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.014+0000 7fbdeb1fd640 1 librados: starting msgr at 2024-03-29T11:27:05.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.014+0000 7fbdeb1fd640 1 librados: starting objecter 2024-03-29T11:27:05.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.014+0000 7fbdeb1fd640 1 Processor -- start 2024-03-29T11:27:05.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.014+0000 7fbdeb1fd640 1 -- start start 2024-03-29T11:27:05.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.014+0000 7fbdeb1fd640 1 librados: setting wanted keys 2024-03-29T11:27:05.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.014+0000 7fbdeb1fd640 1 librados: calling monclient init 2024-03-29T11:27:05.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.014+0000 7fbdeb1fd640 10 monclient: init 2024-03-29T11:27:05.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.015+0000 7fbdeb1fd640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:05.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.015+0000 7fbdeb1fd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:05.006 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.015+0000 7fbdeb1fd640 10 monclient(hunting): picked mon.noname-c con 0x7fbddc035e00 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:05.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.015+0000 7fbdeb1fd640 10 monclient(hunting): picked mon.noname-b con 0x7fbddc0c9670 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:05.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.015+0000 7fbdeb1fd640 10 monclient(hunting): picked mon.noname-a con 0x7fbddc0beed0 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:05.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.015+0000 7fbdeb1fd640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc030f10 con 0x7fbddc0beed0 2024-03-29T11:27:05.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.015+0000 7fbdeb1fd640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc031eb0 con 0x7fbddc0c9670 2024-03-29T11:27:05.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.015+0000 7fbdeb1fd640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc032190 con 0x7fbddc035e00 2024-03-29T11:27:05.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.015+0000 7fbdeb1fd640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:05.007 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.015+0000 7fbdeb1fd640 10 monclient(hunting): authenticate will time out at 1275.129835s 2024-03-29T11:27:05.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.018+0000 7fbdea9fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbddc0beed0 0x7fbddc032fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39525/0 (socket says 172.21.15.59:39525) 2024-03-29T11:27:05.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.018+0000 7fbdea9fc640 1 -- 172.21.15.59:0/2021877954 learned_addr learned my addr 172.21.15.59:0/2021877954 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:05.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.019+0000 7fbddaffd640 10 client.?.objecter ms_handle_connect 0x7fbddc035e00 2024-03-29T11:27:05.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.019+0000 7fbddaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:05.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.019+0000 7fbddaffd640 10 client.?.objecter ms_handle_connect 0x7fbddc0beed0 2024-03-29T11:27:05.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.019+0000 7fbddaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:05.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.019+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268711965 0 0) 0x7fbddc032190 con 0x7fbddc035e00 2024-03-29T11:27:05.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.019+0000 7fbddaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:05.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.019+0000 7fbddaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:05.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.019+0000 7fbddaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:05.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.019+0000 7fbddaffd640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:27:05.009 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.019+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc8004eb0 con 0x7fbddc035e00 2024-03-29T11:27:05.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1462357009 0 0) 0x7fbdc8004eb0 con 0x7fbddc035e00 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdc8005300 con 0x7fbddc035e00 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195101570 0 0) 0x7fbddc030f10 con 0x7fbddc0beed0 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc8004eb0 con 0x7fbddc0beed0 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fbde00017a0 con 0x7fbddc035e00 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): dump: 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:05.013 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 client.?.objecter ms_handle_connect 0x7fbddc0c9670 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877771965 0 0) 0x7fbddc031eb0 con 0x7fbddc0c9670 2024-03-29T11:27:05.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbddc030f10 con 0x7fbddc0c9670 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070795496 0 0) 0x7fbdc8004eb0 con 0x7fbddc0beed0 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdc8005ca0 con 0x7fbddc0beed0 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fbde404f400 con 0x7fbddc0beed0 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.022+0000 7fbddaffd640 10 monclient(hunting): dump: 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:05.015 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4230900882 0 0) 0x7fbdc8005300 con 0x7fbddc035e00 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 >> v1:172.21.15.172:6789/0 conn(0x7fbddc0c9670 legacy=0x7fbddc032890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 >> v1:172.21.15.59:6789/0 conn(0x7fbddc0beed0 legacy=0x7fbddc032fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:05.016 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 1 monclient: found mon.c 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc032470 con 0x7fbddc035e00 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbdeb1fd640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbdeb1fd640 10 monclient: _renew_subs 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbdeb1fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/2021877954 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbddc0313a0 con 0x7fbddc035e00 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbdeb1fd640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:05.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbdeb1fd640 10 monclient: _renew_subs 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbdeb1fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/2021877954 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbddc0cac00 con 0x7fbddc035e00 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbdeb1fd640 1 librados: init done 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbde0001900 con 0x7fbddc035e00 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fbde0015a20 con 0x7fbddc035e00 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbddaffd640 10 monclient: dump: 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:05.018 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:05.019 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:05.019 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:05.019 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:05.019 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:05.019 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:05.019 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:05.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.023+0000 7fbde89f8640 4 set_mon_vals no callback set 2024-03-29T11:27:05.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.025+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fbde002b9f0 con 0x7fbddc035e00 2024-03-29T11:27:05.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.025+0000 7fbdeb1fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:05.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.025+0000 7fbdeb1fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:05.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.025+0000 7fbdeb1fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:05.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.025+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/2021877954 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdcc005080 con 0x7fbddc035e00 2024-03-29T11:27:05.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.026+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fbde0065560 con 0x7fbddc035e00 2024-03-29T11:27:05.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.026+0000 7fbddaffd640 10 client.4511.objecter ms_dispatch 0x7fbde4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:05.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.026+0000 7fbddaffd640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:05.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.026+0000 7fbddaffd640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:05.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.026+0000 7fbddaffd640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T11:27:05.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.027+0000 7fbde99fa640 10 monclient: get_auth_request con 0x7fbdc806e620 auth_method 0 2024-03-29T11:27:05.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.027+0000 7fbddaffd640 10 client.4511.objecter ms_handle_connect 0x7fbdc806e620 2024-03-29T11:27:05.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.035+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbde0061800 con 0x7fbddc035e00 2024-03-29T11:27:05.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.035+0000 7fbddaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:05.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.035+0000 7fbddaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:05.355 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.367+0000 7fbdeb1fd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.356 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.367+0000 7fbdeb1fd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.356 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.367+0000 7fbdeb1fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:05.356 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.367+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/2021877954 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdcc005330 con 0x7fbddc035e00 2024-03-29T11:27:05.356 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.368+0000 7fbddaffd640 1 -- 172.21.15.59:0/2021877954 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fbde0061800 con 0x7fbddc035e00 2024-03-29T11:27:05.356 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.368+0000 7fbddaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.356 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.368+0000 7fbddaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:05.358 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:27:05.358 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:05.360 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.372+0000 7fbdeb1fd640 10 librados: watch_flush enter 2024-03-29T11:27:05.360 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.372+0000 7fbdeb1fd640 10 librados: watch_flush exit 2024-03-29T11:27:05.360 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.372+0000 7fbdeb1fd640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T11:27:05.360 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.372+0000 7fbdeb1fd640 10 client.4511.objecter successfully canceled tick 2024-03-29T11:27:05.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.373+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/2021877954 >> v1:172.21.15.172:6816/2420742773 conn(0x7fbdc806e620 legacy=0x7fbdc8070ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.373+0000 7fbdeb1fd640 10 monclient: shutdown 2024-03-29T11:27:05.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.373+0000 7fbdeb1fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:05.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.373+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/2021877954 >> v1:172.21.15.59:6790/0 conn(0x7fbddc035e00 legacy=0x7fbddc0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.373+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/2021877954 shutdown_connections 2024-03-29T11:27:05.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.373+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/2021877954 >> 172.21.15.59:0/2021877954 conn(0x7fbddc005c00 msgr2=0x7fbddc0c6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:05.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.373+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/2021877954 shutdown_connections 2024-03-29T11:27:05.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.373+0000 7fbdeb1fd640 1 -- 172.21.15.59:0/2021877954 wait complete. 2024-03-29T11:27:05.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:05.374+0000 7fbdeb1fd640 1 librados: shutdown 2024-03-29T11:27:05.382 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:12.191 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:12.342 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.353+0000 7f186ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:27:12.342 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.353+0000 7f186ffff640 10 monclient: build_initial_monmap 2024-03-29T11:27:12.342 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.353+0000 7f186ffff640 10 monclient: monmap: 2024-03-29T11:27:12.342 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:12.342 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:12.342 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:27:12.355361+0000 2024-03-29T11:27:12.342 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:27:12.355361+0000 2024-03-29T11:27:12.342 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:12.342 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:12.342 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:12.343 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:27:12.343 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:27:12.343 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:12.343 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.355+0000 7f186ffff640 1 Processor -- start 2024-03-29T11:27:12.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.355+0000 7f186ffff640 1 -- start start 2024-03-29T11:27:12.345 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.355+0000 7f186ffff640 10 monclient: init 2024-03-29T11:27:12.346 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.356+0000 7f186ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:12.346 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.356+0000 7f186ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:12.347 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.359+0000 7f186ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f18700b4820 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:12.347 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.359+0000 7f186effd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f18700b4820 0x7f187013baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49515/0 (socket says 172.21.15.59:49515) 2024-03-29T11:27:12.348 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.359+0000 7f186effd640 1 -- 172.21.15.59:0/2012489004 learned_addr learned my addr 172.21.15.59:0/2012489004 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:12.348 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.359+0000 7f186ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f18700a9ce0 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:12.351 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.362+0000 7f186ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f18700b3cd0 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:12.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.362+0000 7f186ffff640 1 -- 172.21.15.59:0/2012489004 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18700b4180 con 0x7f18700b4820 2024-03-29T11:27:12.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.362+0000 7f186ffff640 1 -- 172.21.15.59:0/2012489004 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18700aaf60 con 0x7f18700a9ce0 2024-03-29T11:27:12.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.363+0000 7f186ffff640 1 -- 172.21.15.59:0/2012489004 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187013c200 con 0x7f18700b3cd0 2024-03-29T11:27:12.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.363+0000 7f186ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:12.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.363+0000 7f186ffff640 10 monclient(hunting): authenticate will time out at 1282.477923s 2024-03-29T11:27:12.352 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.363+0000 7f186dffb640 1 -- 172.21.15.59:0/2012489004 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340687611 0 0) 0x7f1860001110 con 0x7f18700b4820 2024-03-29T11:27:12.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.364+0000 7f186dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.364+0000 7f186dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.364+0000 7f186dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.364+0000 7f186dffb640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:27:12.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 1 -- 172.21.15.59:0/2012489004 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1858003520 con 0x7f18700b4820 2024-03-29T11:27:12.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 1 -- 172.21.15.59:0/2012489004 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092202704 0 0) 0x7f187013c200 con 0x7f18700b3cd0 2024-03-29T11:27:12.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.361 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:27:12.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 1 -- 172.21.15.59:0/2012489004 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1860001110 con 0x7f18700b3cd0 2024-03-29T11:27:12.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 1 -- 172.21.15.59:0/2012489004 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2147722195 0 0) 0x7f1858003520 con 0x7f18700b4820 2024-03-29T11:27:12.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 1 -- 172.21.15.59:0/2012489004 >> v1:172.21.15.59:6790/0 conn(0x7f18700b3cd0 legacy=0x7f187014d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 1 -- 172.21.15.59:0/2012489004 >> v1:172.21.15.172:6789/0 conn(0x7f18700a9ce0 legacy=0x7f1870145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:12.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:12.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 1 monclient: found mon.noname-a 2024-03-29T11:27:12.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:12.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:27:12.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 1 -- 172.21.15.59:0/2012489004 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1870145e80 con 0x7f18700b4820 2024-03-29T11:27:12.362 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:12.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:12.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:12.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 1 -- 172.21.15.59:0/2012489004 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1860001680 con 0x7f18700b4820 2024-03-29T11:27:12.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:12.363 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186dffb640 10 monclient: dump: 2024-03-29T11:27:12.363 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:12.363 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:12.363 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:12.363 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:12.363 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186ffff640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.365+0000 7f186ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.369+0000 7f186dffb640 1 -- 172.21.15.59:0/2012489004 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18600017e0 con 0x7f18700b4820 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.369+0000 7f186dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.369+0000 7f186dffb640 1 -- 172.21.15.59:0/2012489004 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1860001bf0 con 0x7f18700b4820 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.369+0000 7f186ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.369+0000 7f186ffff640 4 set_mon_vals no callback set 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.370+0000 7f186dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.370+0000 7f186dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:12.364 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.370+0000 7f186dffb640 10 monclient: dump: 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.370+0000 7f186ffff640 10 monclient: shutdown 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.370+0000 7f186ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.370+0000 7f186ffff640 1 -- 172.21.15.59:0/2012489004 >> v1:172.21.15.59:6789/0 conn(0x7f18700b4820 legacy=0x7f187013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.371+0000 7f186ffff640 1 -- 172.21.15.59:0/2012489004 shutdown_connections 2024-03-29T11:27:12.365 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.371+0000 7f186ffff640 1 -- 172.21.15.59:0/2012489004 >> 172.21.15.59:0/2012489004 conn(0x7f187006d070 msgr2=0x7f1870066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:12.367 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.372+0000 7f186ffff640 1 -- 172.21.15.59:0/2012489004 shutdown_connections 2024-03-29T11:27:12.367 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.373+0000 7f186ffff640 1 -- 172.21.15.59:0/2012489004 wait complete. 2024-03-29T11:27:12.367 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.374+0000 7f186ffff640 10 monclient: build_initial_monmap 2024-03-29T11:27:12.367 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.374+0000 7f186ffff640 10 monclient: monmap: 2024-03-29T11:27:12.367 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.375+0000 7f186ffff640 1 librados: starting msgr at 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.375+0000 7f186ffff640 1 librados: starting objecter 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.375+0000 7f186ffff640 1 Processor -- start 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.375+0000 7f186ffff640 1 -- start start 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.375+0000 7f186ffff640 1 librados: setting wanted keys 2024-03-29T11:27:12.368 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.375+0000 7f186ffff640 1 librados: calling monclient init 2024-03-29T11:27:12.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.376+0000 7f186ffff640 10 monclient: init 2024-03-29T11:27:12.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.376+0000 7f186ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:12.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.376+0000 7f186ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:12.369 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.377+0000 7f186ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f18700a9ce0 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:12.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.383+0000 7f186effd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f18700a9ce0 0x7f1870134160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49713/0 (socket says 172.21.15.59:49713) 2024-03-29T11:27:12.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.384+0000 7f186ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f18700b3cd0 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:12.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.384+0000 7f186ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f1870144100 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:12.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.384+0000 7f186ffff640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187013b380 con 0x7f18700a9ce0 2024-03-29T11:27:12.372 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.384+0000 7f186ffff640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18701336c0 con 0x7f18700b3cd0 2024-03-29T11:27:12.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.384+0000 7f186ffff640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187013b5b0 con 0x7f1870144100 2024-03-29T11:27:12.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.384+0000 7f186ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:12.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.384+0000 7f186ffff640 10 monclient(hunting): authenticate will time out at 1282.498852s 2024-03-29T11:27:12.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.384+0000 7f186effd640 1 -- 172.21.15.59:0/1971204227 learned_addr learned my addr 172.21.15.59:0/1971204227 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:12.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.384+0000 7f18577fe640 10 client.?.objecter ms_handle_connect 0x7f18700a9ce0 2024-03-29T11:27:12.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.384+0000 7f18577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:12.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.384+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647127043 0 0) 0x7f187013b380 con 0x7f18700a9ce0 2024-03-29T11:27:12.373 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:27:12.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185c004eb0 con 0x7f18700a9ce0 2024-03-29T11:27:12.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 10 client.?.objecter ms_handle_connect 0x7f18700b3cd0 2024-03-29T11:27:12.374 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:12.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975992372 0 0) 0x7f18701336c0 con 0x7f18700b3cd0 2024-03-29T11:27:12.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:27:12.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.385+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f187013b380 con 0x7f18700b3cd0 2024-03-29T11:27:12.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.386+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1007683831 0 0) 0x7f185c004eb0 con 0x7f18700a9ce0 2024-03-29T11:27:12.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.386+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18701336c0 con 0x7f18700a9ce0 2024-03-29T11:27:12.378 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.386+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f18600047e0 con 0x7f18700a9ce0 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.386+0000 7f18577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.386+0000 7f18577fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.386+0000 7f18577fe640 10 monclient(hunting): dump: 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:12.379 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.386+0000 7f18577fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.387+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 876031763 0 0) 0x7f18701336c0 con 0x7f18700a9ce0 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.387+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 >> v1:172.21.15.59:6790/0 conn(0x7f1870144100 legacy=0x7f1870132fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.387+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 >> v1:172.21.15.172:6789/0 conn(0x7f18700b3cd0 legacy=0x7f18701328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.387+0000 7f18577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.387+0000 7f18577fe640 1 monclient: found mon.a 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.387+0000 7f18577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.387+0000 7f18577fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.387+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187013b890 con 0x7f18700a9ce0 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.387+0000 7f18577fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.387+0000 7f18577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.387+0000 7f18577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f186ffff640 5 monclient: authenticate success, global_id 4506 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f186ffff640 10 monclient: _renew_subs 2024-03-29T11:27:12.380 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f186ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f186ffff640 1 -- 172.21.15.59:0/1971204227 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1870144dd0 con 0x7f18700a9ce0 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f186ffff640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f186ffff640 10 monclient: _renew_subs 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f186ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1860004bf0 con 0x7f18700a9ce0 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f186ffff640 1 -- 172.21.15.59:0/1971204227 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18701449d0 con 0x7f18700a9ce0 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f186ffff640 1 librados: init done 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f18577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1860004f40 con 0x7f18700a9ce0 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f18577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f18577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.388+0000 7f18577fe640 10 monclient: dump: 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:12.381 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:12.382 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:12.382 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:12.382 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.382 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:12.382 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:12.382 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:12.382 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:12.382 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:12.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.390+0000 7f186cff9640 4 set_mon_vals no callback set 2024-03-29T11:27:12.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.391+0000 7f186ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.392+0000 7f186ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.392+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f186001af10 con 0x7f18700a9ce0 2024-03-29T11:27:12.383 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.392+0000 7f186ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:12.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.392+0000 7f186ffff640 1 -- 172.21.15.59:0/1971204227 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1850005080 con 0x7f18700a9ce0 2024-03-29T11:27:12.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.392+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f1860084c30 con 0x7f18700a9ce0 2024-03-29T11:27:12.384 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.392+0000 7f18577fe640 10 client.4506.objecter ms_dispatch 0x7f1870000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:12.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.392+0000 7f18577fe640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:12.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.392+0000 7f18577fe640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:12.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.392+0000 7f18577fe640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-29T11:27:12.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.393+0000 7f186e7fc640 10 monclient: get_auth_request con 0x7f185c06dce0 auth_method 0 2024-03-29T11:27:12.385 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.396+0000 7f18577fe640 10 client.4506.objecter ms_handle_connect 0x7f185c06dce0 2024-03-29T11:27:12.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.401+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1860050570 con 0x7f18700a9ce0 2024-03-29T11:27:12.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.401+0000 7f18577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.389 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.401+0000 7f18577fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:12.724 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.736+0000 7f186ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.724 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.736+0000 7f186ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.725 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.736+0000 7f186ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:12.725 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.736+0000 7f186ffff640 1 -- 172.21.15.59:0/1971204227 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1850005330 con 0x7f18700a9ce0 2024-03-29T11:27:12.727 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.738+0000 7f18577fe640 1 -- 172.21.15.59:0/1971204227 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f18600524a0 con 0x7f18700a9ce0 2024-03-29T11:27:12.727 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.738+0000 7f18577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.727 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.738+0000 7f18577fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:12.728 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:27:12.728 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:12.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.742+0000 7f186ffff640 10 librados: watch_flush enter 2024-03-29T11:27:12.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.743+0000 7f186ffff640 10 librados: watch_flush exit 2024-03-29T11:27:12.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.743+0000 7f186ffff640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-29T11:27:12.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.743+0000 7f186ffff640 10 client.4506.objecter successfully canceled tick 2024-03-29T11:27:12.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.744+0000 7f186ffff640 1 -- 172.21.15.59:0/1971204227 >> v1:172.21.15.172:6816/2420742773 conn(0x7f185c06dce0 legacy=0x7f185c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.744+0000 7f186ffff640 10 monclient: shutdown 2024-03-29T11:27:12.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.744+0000 7f186ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:12.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.744+0000 7f186ffff640 1 -- 172.21.15.59:0/1971204227 >> v1:172.21.15.59:6789/0 conn(0x7f18700a9ce0 legacy=0x7f1870134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.744+0000 7f186ffff640 1 -- 172.21.15.59:0/1971204227 shutdown_connections 2024-03-29T11:27:12.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.744+0000 7f186ffff640 1 -- 172.21.15.59:0/1971204227 >> 172.21.15.59:0/1971204227 conn(0x7f187006d070 msgr2=0x7f18700b72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:12.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.744+0000 7f186ffff640 1 -- 172.21.15.59:0/1971204227 shutdown_connections 2024-03-29T11:27:12.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.744+0000 7f186ffff640 1 -- 172.21.15.59:0/1971204227 wait complete. 2024-03-29T11:27:12.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:12.745+0000 7f186ffff640 1 librados: shutdown 2024-03-29T11:27:12.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:19.565 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:19.715 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.723+0000 7fc34394b640 10 monclient: get_monmap_and_config 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.723+0000 7fc34394b640 10 monclient: build_initial_monmap 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.723+0000 7fc34394b640 10 monclient: monmap: 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:27:19.725650+0000 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:27:19.725650+0000 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:19.716 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.725+0000 7fc34394b640 1 Processor -- start 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.725+0000 7fc34394b640 1 -- start start 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.725+0000 7fc34394b640 10 monclient: init 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.726+0000 7fc34394b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.726+0000 7fc34394b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.726+0000 7fc34394b640 10 monclient(hunting): picked mon.noname-b con 0x7fc33c1422b0 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc34394b640 10 monclient(hunting): picked mon.noname-c con 0x7fc33c142690 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc34394b640 10 monclient(hunting): picked mon.noname-a con 0x7fc33c0b17d0 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc34394b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c142c80 con 0x7fc33c0b17d0 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc34394b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c0aae40 con 0x7fc33c142690 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc34394b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c13fbd0 con 0x7fc33c1422b0 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc34394b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc34394b640 10 monclient(hunting): authenticate will time out at 1289.841788s 2024-03-29T11:27:19.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc3416e7640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fc33c1422b0 0x7fc33c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:38160/0 (socket says 172.21.15.59:38160) 2024-03-29T11:27:19.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc3416e7640 1 -- 172.21.15.59:0/11120275 learned_addr learned my addr 172.21.15.59:0/11120275 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc333fff640 1 -- 172.21.15.59:0/11120275 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144071121 0 0) 0x7fc33c13fbd0 con 0x7fc33c1422b0 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc333fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc333fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc333fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc333fff640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.727+0000 7fc333fff640 1 -- 172.21.15.59:0/11120275 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc328003520 con 0x7fc33c1422b0 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 1 -- 172.21.15.59:0/11120275 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576067337 0 0) 0x7fc33c142c80 con 0x7fc33c0b17d0 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 1 -- 172.21.15.59:0/11120275 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc33c13fbd0 con 0x7fc33c0b17d0 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 1 -- 172.21.15.59:0/11120275 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2846560499 0 0) 0x7fc328003520 con 0x7fc33c1422b0 2024-03-29T11:27:19.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 1 -- 172.21.15.59:0/11120275 >> v1:172.21.15.172:6789/0 conn(0x7fc33c142690 legacy=0x7fc33c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 1 -- 172.21.15.59:0/11120275 >> v1:172.21.15.59:6789/0 conn(0x7fc33c0b17d0 legacy=0x7fc33c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 1 monclient: found mon.noname-b 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 1 -- 172.21.15.59:0/11120275 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c151940 con 0x7fc33c1422b0 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc34394b640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc34394b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 1 -- 172.21.15.59:0/11120275 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc338001610 con 0x7fc33c1422b0 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:19.720 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient: dump: 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc34394b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 1 -- 172.21.15.59:0/11120275 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc338001770 con 0x7fc33c1422b0 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 1 -- 172.21.15.59:0/11120275 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc338001bc0 con 0x7fc33c1422b0 2024-03-29T11:27:19.721 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc333fff640 10 monclient: dump: 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc34394b640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.728+0000 7fc34394b640 4 set_mon_vals no callback set 2024-03-29T11:27:19.722 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.729+0000 7fc34394b640 10 monclient: shutdown 2024-03-29T11:27:19.723 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.729+0000 7fc34394b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:19.723 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.729+0000 7fc34394b640 1 -- 172.21.15.59:0/11120275 >> v1:172.21.15.59:6790/0 conn(0x7fc33c1422b0 legacy=0x7fc33c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.723 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.730+0000 7fc34394b640 1 -- 172.21.15.59:0/11120275 shutdown_connections 2024-03-29T11:27:19.723 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.730+0000 7fc34394b640 1 -- 172.21.15.59:0/11120275 >> 172.21.15.59:0/11120275 conn(0x7fc33c066a00 msgr2=0x7fc33c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:19.723 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.730+0000 7fc34394b640 1 -- 172.21.15.59:0/11120275 shutdown_connections 2024-03-29T11:27:19.723 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.730+0000 7fc34394b640 1 -- 172.21.15.59:0/11120275 wait complete. 2024-03-29T11:27:19.727 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.731+0000 7fc34394b640 10 monclient: build_initial_monmap 2024-03-29T11:27:19.727 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.731+0000 7fc34394b640 10 monclient: monmap: 2024-03-29T11:27:19.727 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:19.727 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:19.727 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.732+0000 7fc34394b640 1 librados: starting msgr at 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.732+0000 7fc34394b640 1 librados: starting objecter 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.732+0000 7fc34394b640 1 Processor -- start 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.732+0000 7fc34394b640 1 -- start start 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.732+0000 7fc34394b640 1 librados: setting wanted keys 2024-03-29T11:27:19.728 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.732+0000 7fc34394b640 1 librados: calling monclient init 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.732+0000 7fc34394b640 10 monclient: init 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.733+0000 7fc34394b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.733+0000 7fc34394b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.733+0000 7fc34394b640 10 monclient(hunting): picked mon.noname-a con 0x7fc33c0b17d0 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.733+0000 7fc3416e7640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc33c0b17d0 0x7fc33c1ded30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38030/0 (socket says 172.21.15.59:38030) 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.733+0000 7fc3416e7640 1 -- 172.21.15.59:0/3141901144 learned_addr learned my addr 172.21.15.59:0/3141901144 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.733+0000 7fc34394b640 10 monclient(hunting): picked mon.noname-b con 0x7fc33c1422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.734+0000 7fc331ffb640 10 client.?.objecter ms_handle_connect 0x7fc33c0b17d0 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.734+0000 7fc331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.734+0000 7fc331ffb640 10 client.?.objecter ms_handle_connect 0x7fc33c1422b0 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.734+0000 7fc331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.734+0000 7fc34394b640 10 monclient(hunting): picked mon.noname-c con 0x7fc33c142690 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.734+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c132100 con 0x7fc33c0b17d0 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.734+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c06e660 con 0x7fc33c1422b0 2024-03-29T11:27:19.729 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.734+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c06e8d0 con 0x7fc33c142690 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.734+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267035264 0 0) 0x7fc33c132100 con 0x7fc33c0b17d0 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.734+0000 7fc34394b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.734+0000 7fc34394b640 10 monclient(hunting): authenticate will time out at 1289.849639s 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc320004eb0 con 0x7fc33c0b17d0 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 client.?.objecter ms_handle_connect 0x7fc33c142690 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912067101 0 0) 0x7fc33c06e660 con 0x7fc33c1422b0 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.730 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc33c132100 con 0x7fc33c1422b0 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938705684 0 0) 0x7fc33c06e8d0 con 0x7fc33c142690 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc33c06e660 con 0x7fc33c142690 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1004396714 0 0) 0x7fc320004eb0 con 0x7fc33c0b17d0 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc33c06e8d0 con 0x7fc33c0b17d0 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc338001b80 con 0x7fc33c0b17d0 2024-03-29T11:27:19.731 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:19.732 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.735+0000 7fc331ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3716737406 0 0) 0x7fc33c06e660 con 0x7fc33c142690 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc320004eb0 con 0x7fc33c142690 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc334001c30 con 0x7fc33c142690 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:19.733 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 438651352 0 0) 0x7fc33c06e8d0 con 0x7fc33c0b17d0 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 >> v1:172.21.15.59:6790/0 conn(0x7fc33c142690 legacy=0x7fc33c12e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 >> v1:172.21.15.172:6789/0 conn(0x7fc33c1422b0 legacy=0x7fc33c12d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 1 monclient: found mon.a 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c131270 con 0x7fc33c0b17d0 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:19.734 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc34394b640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc34394b640 10 monclient: _renew_subs 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc34394b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33c12e790 con 0x7fc33c0b17d0 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc34394b640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc34394b640 10 monclient: _renew_subs 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc34394b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33c132590 con 0x7fc33c0b17d0 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc338004a90 con 0x7fc33c0b17d0 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc338004da0 con 0x7fc33c0b17d0 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc331ffb640 10 monclient: dump: 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:19.735 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:19.736 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:19.736 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:19.736 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.736 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:19.736 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:19.736 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:19.736 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:19.736 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:19.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc34394b640 1 librados: init done 2024-03-29T11:27:19.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.736+0000 7fc332ffd640 4 set_mon_vals no callback set 2024-03-29T11:27:19.736 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.738+0000 7fc34394b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:19.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.738+0000 7fc34394b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:19.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.738+0000 7fc34394b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:19.737 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.738+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33c075f10 con 0x7fc33c0b17d0 2024-03-29T11:27:19.738 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.739+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fc33801ad70 con 0x7fc33c0b17d0 2024-03-29T11:27:19.738 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.740+0000 7fc340ee6640 10 monclient: get_auth_request con 0x7fc32006dfb0 auth_method 0 2024-03-29T11:27:19.738 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.741+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fc338084c00 con 0x7fc33c0b17d0 2024-03-29T11:27:19.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.741+0000 7fc331ffb640 10 client.4512.objecter ms_dispatch 0x7fc33c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:19.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.741+0000 7fc331ffb640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:19.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.741+0000 7fc331ffb640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:19.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.741+0000 7fc331ffb640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-29T11:27:19.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.741+0000 7fc331ffb640 10 client.4512.objecter ms_handle_connect 0x7fc32006dfb0 2024-03-29T11:27:19.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.748+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3380504e0 con 0x7fc33c0b17d0 2024-03-29T11:27:19.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.748+0000 7fc331ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:19.739 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:19.748+0000 7fc331ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:20.089 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.100+0000 7fc34394b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:20.089 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.100+0000 7fc34394b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:20.089 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.100+0000 7fc34394b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:20.089 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.100+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc33c0b2a30 con 0x7fc33c0b17d0 2024-03-29T11:27:20.091 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.103+0000 7fc331ffb640 1 -- 172.21.15.59:0/3141901144 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fc338052410 con 0x7fc33c0b17d0 2024-03-29T11:27:20.091 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.103+0000 7fc331ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:20.091 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.103+0000 7fc331ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:20.092 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:27:20.092 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:20.095 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.106+0000 7fc34394b640 10 librados: watch_flush enter 2024-03-29T11:27:20.096 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.107+0000 7fc34394b640 10 librados: watch_flush exit 2024-03-29T11:27:20.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.107+0000 7fc34394b640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-29T11:27:20.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.107+0000 7fc34394b640 10 client.4512.objecter successfully canceled tick 2024-03-29T11:27:20.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.108+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 >> v1:172.21.15.172:6816/2420742773 conn(0x7fc32006dfb0 legacy=0x7fc320070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.108+0000 7fc34394b640 10 monclient: shutdown 2024-03-29T11:27:20.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.108+0000 7fc34394b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:20.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.108+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 >> v1:172.21.15.59:6789/0 conn(0x7fc33c0b17d0 legacy=0x7fc33c1ded30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.108+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 shutdown_connections 2024-03-29T11:27:20.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.108+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 >> 172.21.15.59:0/3141901144 conn(0x7fc33c066a00 msgr2=0x7fc33c13c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:20.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.110+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 shutdown_connections 2024-03-29T11:27:20.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.110+0000 7fc34394b640 1 -- 172.21.15.59:0/3141901144 wait complete. 2024-03-29T11:27:20.099 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:20.111+0000 7fc34394b640 1 librados: shutdown 2024-03-29T11:27:20.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:26.925 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:27.085 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.097+0000 7f0ddda14640 10 monclient: get_monmap_and_config 2024-03-29T11:27:27.085 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.097+0000 7f0ddda14640 10 monclient: build_initial_monmap 2024-03-29T11:27:27.085 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.097+0000 7f0ddda14640 10 monclient: monmap: 2024-03-29T11:27:27.085 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:27.086 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:27.086 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:27:27.098241+0000 2024-03-29T11:27:27.086 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:27:27.098241+0000 2024-03-29T11:27:27.086 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:27.086 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:27.086 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:27.086 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:27:27.086 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:27:27.086 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:27.089 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.098+0000 7f0ddda14640 1 Processor -- start 2024-03-29T11:27:27.089 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.098+0000 7f0ddda14640 1 -- start start 2024-03-29T11:27:27.089 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.098+0000 7f0ddda14640 10 monclient: init 2024-03-29T11:27:27.090 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0ddda14640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:27.090 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0ddda14640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:27.090 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0ddda14640 10 monclient(hunting): picked mon.noname-c con 0x7f0dd81457f0 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:27.090 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0dd67fc640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f0dd813f650 0x7f0dd80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:53818/0 (socket says 172.21.15.59:53818) 2024-03-29T11:27:27.090 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0dd67fc640 1 -- 172.21.15.59:0/1838598287 learned_addr learned my addr 172.21.15.59:0/1838598287 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:27.090 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0ddda14640 10 monclient(hunting): picked mon.noname-b con 0x7f0dd813f650 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:27.090 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0ddda14640 10 monclient(hunting): picked mon.noname-a con 0x7f0dd80a9d70 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:27.090 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0ddda14640 1 -- 172.21.15.59:0/1838598287 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd80aaef0 con 0x7f0dd80a9d70 2024-03-29T11:27:27.091 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0ddda14640 1 -- 172.21.15.59:0/1838598287 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd8145bd0 con 0x7f0dd81457f0 2024-03-29T11:27:27.091 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0ddda14640 1 -- 172.21.15.59:0/1838598287 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd8145eb0 con 0x7f0dd813f650 2024-03-29T11:27:27.091 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0ddda14640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:27.091 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.099+0000 7f0ddda14640 10 monclient(hunting): authenticate will time out at 1297.213282s 2024-03-29T11:27:27.091 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530499337 0 0) 0x7f0dd8145eb0 con 0x7f0dd813f650 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc0003520 con 0x7f0dd813f650 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882122209 0 0) 0x7f0dd8145bd0 con 0x7f0dd81457f0 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.100+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dd8145eb0 con 0x7f0dd81457f0 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252195052 0 0) 0x7f0dd80aaef0 con 0x7f0dd80a9d70 2024-03-29T11:27:27.092 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc00039d0 con 0x7f0dd80a9d70 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4107246913 0 0) 0x7f0dd8145eb0 con 0x7f0dd81457f0 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 >> v1:172.21.15.59:6790/0 conn(0x7f0dd813f650 legacy=0x7f0dd80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 >> v1:172.21.15.59:6789/0 conn(0x7f0dd80a9d70 legacy=0x7f0dd814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd8146340 con 0x7f0dd81457f0 2024-03-29T11:27:27.093 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0ddda14640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0ddda14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0dd0001970 con 0x7f0dd81457f0 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0dd5ffb640 10 monclient: dump: 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:27.094 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.101+0000 7f0ddda14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.102+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dd0001b50 con 0x7f0dd81457f0 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.102+0000 7f0dd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.102+0000 7f0dd5ffb640 1 -- 172.21.15.59:0/1838598287 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0dd0001970 con 0x7f0dd81457f0 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.102+0000 7f0dd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.102+0000 7f0dd5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.102+0000 7f0dd5ffb640 10 monclient: dump: 2024-03-29T11:27:27.095 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:27.096 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:27.096 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:27.096 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:27.096 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.096 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:27.096 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:27.097 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:27.097 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:27.097 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:27.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.102+0000 7f0ddda14640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:27.097 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.102+0000 7f0ddda14640 4 set_mon_vals no callback set 2024-03-29T11:27:27.110 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.102+0000 7f0ddda14640 10 monclient: shutdown 2024-03-29T11:27:27.110 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.102+0000 7f0ddda14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:27.110 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.102+0000 7f0ddda14640 1 -- 172.21.15.59:0/1838598287 >> v1:172.21.15.172:6789/0 conn(0x7f0dd81457f0 legacy=0x7f0dd813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.110 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.103+0000 7f0ddda14640 1 -- 172.21.15.59:0/1838598287 shutdown_connections 2024-03-29T11:27:27.110 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.103+0000 7f0ddda14640 1 -- 172.21.15.59:0/1838598287 >> 172.21.15.59:0/1838598287 conn(0x7f0dd8066a00 msgr2=0x7f0dd8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:27.110 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.103+0000 7f0ddda14640 1 -- 172.21.15.59:0/1838598287 shutdown_connections 2024-03-29T11:27:27.110 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.103+0000 7f0ddda14640 1 -- 172.21.15.59:0/1838598287 wait complete. 2024-03-29T11:27:27.110 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.104+0000 7f0ddda14640 10 monclient: build_initial_monmap 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.104+0000 7f0ddda14640 10 monclient: monmap: 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.104+0000 7f0ddda14640 1 librados: starting msgr at 2024-03-29T11:27:27.111 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.104+0000 7f0ddda14640 1 librados: starting objecter 2024-03-29T11:27:27.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.104+0000 7f0ddda14640 1 Processor -- start 2024-03-29T11:27:27.112 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.104+0000 7f0ddda14640 1 -- start start 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.105+0000 7f0ddda14640 1 librados: setting wanted keys 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.105+0000 7f0ddda14640 1 librados: calling monclient init 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.105+0000 7f0ddda14640 10 monclient: init 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.106+0000 7f0ddda14640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.106+0000 7f0ddda14640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.106+0000 7f0ddda14640 10 monclient(hunting): picked mon.noname-b con 0x7f0dd80a9d70 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.106+0000 7f0ddda14640 10 monclient(hunting): picked mon.noname-c con 0x7f0dd813f650 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.106+0000 7f0ddda14640 10 monclient(hunting): picked mon.noname-a con 0x7f0dd81457f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.106+0000 7f0ddda14640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd8141e50 con 0x7f0dd81457f0 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.106+0000 7f0ddda14640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd806e240 con 0x7f0dd80a9d70 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.106+0000 7f0ddda14640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd8142de0 con 0x7f0dd813f650 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.106+0000 7f0dd67fc640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f0dd813f650 0x7f0dd81ded50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:54486/0 (socket says 172.21.15.59:54486) 2024-03-29T11:27:27.113 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.106+0000 7f0dd67fc640 1 -- 172.21.15.59:0/21908997 learned_addr learned my addr 172.21.15.59:0/21908997 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0ddda14640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0ddda14640 10 monclient(hunting): authenticate will time out at 1297.220908s 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 10 client.?.objecter ms_handle_connect 0x7f0dd813f650 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 10 client.?.objecter ms_handle_connect 0x7f0dd81457f0 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745877600 0 0) 0x7f0dd8142de0 con 0x7f0dd813f650 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc4004eb0 con 0x7f0dd813f650 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612551459 0 0) 0x7f0dd8141e50 con 0x7f0dd81457f0 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.107+0000 7f0dbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.108+0000 7f0dbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.108+0000 7f0dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.108+0000 7f0dbffff640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.108+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dd8142de0 con 0x7f0dd81457f0 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.108+0000 7f0dbffff640 10 client.?.objecter ms_handle_connect 0x7f0dd80a9d70 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.108+0000 7f0dbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.108+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681493943 0 0) 0x7f0dd806e240 con 0x7f0dd80a9d70 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dd8141e50 con 0x7f0dd80a9d70 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3007098660 0 0) 0x7f0dd8142de0 con 0x7f0dd81457f0 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dd806e240 con 0x7f0dd81457f0 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0dcc0018b0 con 0x7f0dd81457f0 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 10 monclient(hunting): dump: 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:27.116 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1858494002 0 0) 0x7f0dc4004eb0 con 0x7f0dd813f650 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.109+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dd8142de0 con 0x7f0dd813f650 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0dc8001950 con 0x7f0dd813f650 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 10 monclient(hunting): dump: 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:27.117 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:27.118 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:27.118 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:27.118 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:27.118 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 300703547 0 0) 0x7f0dd806e240 con 0x7f0dd81457f0 2024-03-29T11:27:27.118 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 >> v1:172.21.15.59:6790/0 conn(0x7f0dd813f650 legacy=0x7f0dd81ded50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.118 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 >> v1:172.21.15.172:6789/0 conn(0x7f0dd80a9d70 legacy=0x7f0dd8140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.118 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:27.118 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 1 monclient: found mon.a 2024-03-29T11:27:27.118 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:27.118 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:27.118 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd81422e0 con 0x7f0dd81457f0 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 10 monclient: _finish_auth 0 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0dbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0ddda14640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0ddda14640 10 monclient: _renew_subs 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0ddda14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0ddda14640 1 -- 172.21.15.59:0/21908997 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd8141810 con 0x7f0dd81457f0 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0ddda14640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0ddda14640 10 monclient: _renew_subs 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.110+0000 7f0ddda14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.111+0000 7f0ddda14640 1 -- 172.21.15.59:0/21908997 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd81df640 con 0x7f0dd81457f0 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.111+0000 7f0ddda14640 1 librados: init done 2024-03-29T11:27:27.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.111+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dcc0035f0 con 0x7f0dd81457f0 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.111+0000 7f0dbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.111+0000 7f0dd57fa640 4 set_mon_vals no callback set 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.111+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0dcc003900 con 0x7f0dd81457f0 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.111+0000 7f0dbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.111+0000 7f0dbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.111+0000 7f0dbffff640 10 monclient: dump: 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:27.120 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.112+0000 7f0ddda14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.112+0000 7f0ddda14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.112+0000 7f0ddda14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.112+0000 7f0ddda14640 1 -- 172.21.15.59:0/21908997 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db8005080 con 0x7f0dd81457f0 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.113+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f0dcc0198d0 con 0x7f0dd81457f0 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.113+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f0dcc083810 con 0x7f0dd81457f0 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.113+0000 7f0dbffff640 10 client.4518.objecter ms_dispatch 0x7f0dd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.114+0000 7f0dd6ffd640 10 monclient: get_auth_request con 0x7f0dc406ddd0 auth_method 0 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.114+0000 7f0dbffff640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.114+0000 7f0dbffff640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.114+0000 7f0dbffff640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-29T11:27:27.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.114+0000 7f0dbffff640 10 client.4518.objecter ms_handle_connect 0x7f0dc406ddd0 2024-03-29T11:27:27.123 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.122+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0dcc04ef20 con 0x7f0dd81457f0 2024-03-29T11:27:27.123 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.123+0000 7f0dbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:27.123 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.123+0000 7f0dbffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:27.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.500+0000 7f0ddda14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.500+0000 7f0ddda14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.500+0000 7f0ddda14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:27:27.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.500+0000 7f0ddda14640 1 -- 172.21.15.59:0/21908997 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0db8005330 con 0x7f0dd81457f0 2024-03-29T11:27:27.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.502+0000 7f0dbffff640 1 -- 172.21.15.59:0/21908997 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f0dcc051020 con 0x7f0dd81457f0 2024-03-29T11:27:27.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.502+0000 7f0dbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.502+0000 7f0dbffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:27.490 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:27:27.490 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:27.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.505+0000 7f0ddda14640 10 librados: watch_flush enter 2024-03-29T11:27:27.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.505+0000 7f0ddda14640 10 librados: watch_flush exit 2024-03-29T11:27:27.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.505+0000 7f0ddda14640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-29T11:27:27.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.505+0000 7f0ddda14640 10 client.4518.objecter successfully canceled tick 2024-03-29T11:27:27.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.506+0000 7f0ddda14640 1 -- 172.21.15.59:0/21908997 >> v1:172.21.15.172:6816/2420742773 conn(0x7f0dc406ddd0 legacy=0x7f0dc4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.506+0000 7f0ddda14640 10 monclient: shutdown 2024-03-29T11:27:27.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.506+0000 7f0ddda14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:27.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.506+0000 7f0ddda14640 1 -- 172.21.15.59:0/21908997 >> v1:172.21.15.59:6789/0 conn(0x7f0dd81457f0 legacy=0x7f0dd81e62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.506+0000 7f0ddda14640 1 -- 172.21.15.59:0/21908997 shutdown_connections 2024-03-29T11:27:27.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.506+0000 7f0ddda14640 1 -- 172.21.15.59:0/21908997 >> 172.21.15.59:0/21908997 conn(0x7f0dd8066a00 msgr2=0x7f0dd8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:27.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.506+0000 7f0ddda14640 1 -- 172.21.15.59:0/21908997 shutdown_connections 2024-03-29T11:27:27.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.506+0000 7f0ddda14640 1 -- 172.21.15.59:0/21908997 wait complete. 2024-03-29T11:27:27.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:27.506+0000 7f0ddda14640 1 librados: shutdown 2024-03-29T11:27:27.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:34.319 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:34.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.479+0000 7f237e6b3640 10 monclient: get_monmap_and_config 2024-03-29T11:27:34.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.479+0000 7f237e6b3640 10 monclient: build_initial_monmap 2024-03-29T11:27:34.467 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.479+0000 7f237e6b3640 10 monclient: monmap: 2024-03-29T11:27:34.467 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:34.468 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:34.468 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:27:34.480299+0000 2024-03-29T11:27:34.468 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:27:34.480299+0000 2024-03-29T11:27:34.468 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:34.468 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:34.468 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:34.468 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:27:34.468 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:27:34.468 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.480+0000 7f237e6b3640 1 Processor -- start 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.480+0000 7f237e6b3640 1 -- start start 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.480+0000 7f237e6b3640 10 monclient: init 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f237e6b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f237e6b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f237e6b3640 10 monclient(hunting): picked mon.noname-c con 0x7f23781422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f237e6b3640 10 monclient(hunting): picked mon.noname-a con 0x7f2378142690 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f237e6b3640 10 monclient(hunting): picked mon.noname-b con 0x7f23780b17d0 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f237e6b3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2378142c80 con 0x7f2378142690 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f237e6b3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23780aae40 con 0x7f23781422b0 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f237e6b3640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237813fbd0 con 0x7f23780b17d0 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f237e6b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:34.471 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f237e6b3640 10 monclient(hunting): authenticate will time out at 1304.594996s 2024-03-29T11:27:34.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f2377fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f23781422b0 0x7f237813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:55584/0 (socket says 172.21.15.59:55584) 2024-03-29T11:27:34.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f2377fff640 1 -- 172.21.15.59:0/3868366048 learned_addr learned my addr 172.21.15.59:0/3868366048 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:34.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.481+0000 7f2376ffd640 1 -- 172.21.15.59:0/3868366048 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242949658 0 0) 0x7f23780aae40 con 0x7f23781422b0 2024-03-29T11:27:34.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.482+0000 7f2376ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.482+0000 7f2376ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.482+0000 7f2376ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.482+0000 7f2376ffd640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:27:34.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.482+0000 7f2376ffd640 1 -- 172.21.15.59:0/3868366048 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2360003520 con 0x7f23781422b0 2024-03-29T11:27:34.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.482+0000 7f2376ffd640 1 -- 172.21.15.59:0/3868366048 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 610025063 0 0) 0x7f2360003520 con 0x7f23781422b0 2024-03-29T11:27:34.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.482+0000 7f2376ffd640 1 -- 172.21.15.59:0/3868366048 >> v1:172.21.15.59:6790/0 conn(0x7f23780b17d0 legacy=0x7f23780b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.472 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.482+0000 7f2376ffd640 1 -- 172.21.15.59:0/3868366048 >> v1:172.21.15.59:6789/0 conn(0x7f2378142690 legacy=0x7f23780b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:34.473 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:34.473 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 1 monclient: found mon.noname-c 2024-03-29T11:27:34.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:34.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:27:34.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 1 -- 172.21.15.59:0/3868366048 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2378151940 con 0x7f23781422b0 2024-03-29T11:27:34.474 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:34.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:34.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:34.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 1 -- 172.21.15.59:0/3868366048 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f23680015a0 con 0x7f23781422b0 2024-03-29T11:27:34.475 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:34.476 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f2376ffd640 10 monclient: dump: 2024-03-29T11:27:34.476 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:34.476 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:34.476 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:34.476 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:34.476 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.477 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:34.477 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:34.477 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:34.477 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:34.477 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:34.477 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f237e6b3640 5 monclient: authenticate success, global_id 4507 2024-03-29T11:27:34.477 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.485+0000 7f237e6b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:34.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.487+0000 7f2376ffd640 1 -- 172.21.15.59:0/3868366048 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23680017c0 con 0x7f23781422b0 2024-03-29T11:27:34.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.487+0000 7f2376ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:34.485 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.487+0000 7f2376ffd640 1 -- 172.21.15.59:0/3868366048 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2368001b90 con 0x7f23781422b0 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.487+0000 7f2376ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.487+0000 7f2376ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.487+0000 7f2376ffd640 10 monclient: dump: 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:34.486 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.487+0000 7f237e6b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.487+0000 7f237e6b3640 4 set_mon_vals no callback set 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.487+0000 7f237e6b3640 10 monclient: shutdown 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.487+0000 7f237e6b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.487+0000 7f237e6b3640 1 -- 172.21.15.59:0/3868366048 >> v1:172.21.15.172:6789/0 conn(0x7f23781422b0 legacy=0x7f237813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.488+0000 7f237e6b3640 1 -- 172.21.15.59:0/3868366048 shutdown_connections 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.488+0000 7f237e6b3640 1 -- 172.21.15.59:0/3868366048 >> 172.21.15.59:0/3868366048 conn(0x7f2378066a00 msgr2=0x7f2378066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.488+0000 7f237e6b3640 1 -- 172.21.15.59:0/3868366048 shutdown_connections 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.488+0000 7f237e6b3640 1 -- 172.21.15.59:0/3868366048 wait complete. 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.488+0000 7f237e6b3640 10 monclient: build_initial_monmap 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.488+0000 7f237e6b3640 10 monclient: monmap: 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:27:34.487 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.489+0000 7f237e6b3640 1 librados: starting msgr at 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.489+0000 7f237e6b3640 1 librados: starting objecter 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.489+0000 7f237e6b3640 1 Processor -- start 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.489+0000 7f237e6b3640 1 -- start start 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.489+0000 7f237e6b3640 1 librados: setting wanted keys 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.489+0000 7f237e6b3640 1 librados: calling monclient init 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.489+0000 7f237e6b3640 10 monclient: init 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237e6b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:34.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237e6b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237e6b3640 10 monclient(hunting): picked mon.noname-a con 0x7f23780b17d0 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237e6b3640 10 monclient(hunting): picked mon.noname-b con 0x7f23781422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237e6b3640 10 monclient(hunting): picked mon.noname-c con 0x7f2378142690 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237e6b3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2378131a80 con 0x7f23780b17d0 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237e6b3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237806eda0 con 0x7f23781422b0 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237cc50640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f2378142690 0x7f23781e6350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:51164/0 (socket says 172.21.15.59:51164) 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237cc50640 1 -- 172.21.15.59:0/3818505359 learned_addr learned my addr 172.21.15.59:0/3818505359 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237e6b3640 1 -- 172.21.15.59:0/3818505359 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237806f080 con 0x7f2378142690 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f2374ff9640 10 client.?.objecter ms_handle_connect 0x7f2378142690 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f2374ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237e6b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.490+0000 7f237e6b3640 10 monclient(hunting): authenticate will time out at 1304.604530s 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725108408 0 0) 0x7f237806f080 con 0x7f2378142690 2024-03-29T11:27:34.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2364004eb0 con 0x7f2378142690 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 10 client.?.objecter ms_handle_connect 0x7f23781422b0 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3169801665 0 0) 0x7f2364004eb0 con 0x7f2378142690 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f237806f080 con 0x7f2378142690 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2370001d10 con 0x7f2378142690 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.491+0000 7f2374ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:34.490 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.492+0000 7f2374ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.492+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1407517028 0 0) 0x7f237806f080 con 0x7f2378142690 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.492+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 >> v1:172.21.15.172:6789/0 conn(0x7f23781422b0 legacy=0x7f23781df520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.492+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 >> v1:172.21.15.59:6789/0 conn(0x7f23780b17d0 legacy=0x7f23781dee10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 1 monclient: found mon.c 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:34.491 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2378131710 con 0x7f2378142690 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f237e6b3640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f237e6b3640 10 monclient: _renew_subs 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f237e6b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f237e6b3640 1 -- 172.21.15.59:0/3818505359 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237806de90 con 0x7f2378142690 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f237e6b3640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f237e6b3640 10 monclient: _renew_subs 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f237e6b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f237e6b3640 1 -- 172.21.15.59:0/3818505359 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f237813bca0 con 0x7f2378142690 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f237e6b3640 1 librados: init done 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2370001e70 con 0x7f2378142690 2024-03-29T11:27:34.492 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2370003de0 con 0x7f2378142690 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.493+0000 7f2374ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.494+0000 7f2374ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.494+0000 7f2374ff9640 10 monclient: dump: 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:34.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.494+0000 7f23767fc640 4 set_mon_vals no callback set 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.495+0000 7f237e6b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.495+0000 7f237e6b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.495+0000 7f237e6b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.495+0000 7f237e6b3640 1 -- 172.21.15.59:0/3818505359 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2358005080 con 0x7f2378142690 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.496+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f2370003f40 con 0x7f2378142690 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.496+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f2370084e90 con 0x7f2378142690 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.496+0000 7f2374ff9640 10 client.4532.objecter ms_dispatch 0x7f2378000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.496+0000 7f2374ff9640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.496+0000 7f2374ff9640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.496+0000 7f2374ff9640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.497+0000 7f2377fff640 10 monclient: get_auth_request con 0x7f236406dcb0 auth_method 0 2024-03-29T11:27:34.494 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.498+0000 7f2374ff9640 10 client.4532.objecter ms_handle_connect 0x7f236406dcb0 2024-03-29T11:27:34.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.503+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2370050770 con 0x7f2378142690 2024-03-29T11:27:34.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.503+0000 7f2374ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.503+0000 7f2374ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:34.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.821+0000 7f237e6b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:34.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.821+0000 7f237e6b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:34.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.821+0000 7f237e6b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:34.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.821+0000 7f237e6b3640 1 -- 172.21.15.59:0/3818505359 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2358005330 con 0x7f2378142690 2024-03-29T11:27:34.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.822+0000 7f2374ff9640 1 -- 172.21.15.59:0/3818505359 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f23700526a0 con 0x7f2378142690 2024-03-29T11:27:34.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.822+0000 7f2374ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:34.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.822+0000 7f2374ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:34.811 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:27:34.811 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:34.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.826+0000 7f237e6b3640 10 librados: watch_flush enter 2024-03-29T11:27:34.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.826+0000 7f237e6b3640 10 librados: watch_flush exit 2024-03-29T11:27:34.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.826+0000 7f237e6b3640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T11:27:34.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.826+0000 7f237e6b3640 10 client.4532.objecter successfully canceled tick 2024-03-29T11:27:34.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.826+0000 7f237e6b3640 1 -- 172.21.15.59:0/3818505359 >> v1:172.21.15.172:6816/2420742773 conn(0x7f236406dcb0 legacy=0x7f2364070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.826+0000 7f237e6b3640 10 monclient: shutdown 2024-03-29T11:27:34.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.826+0000 7f237e6b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:34.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.826+0000 7f237e6b3640 1 -- 172.21.15.59:0/3818505359 >> v1:172.21.15.59:6790/0 conn(0x7f2378142690 legacy=0x7f23781e6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.827+0000 7f237e6b3640 1 -- 172.21.15.59:0/3818505359 shutdown_connections 2024-03-29T11:27:34.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.827+0000 7f237e6b3640 1 -- 172.21.15.59:0/3818505359 >> 172.21.15.59:0/3818505359 conn(0x7f2378066a00 msgr2=0x7f237813de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:34.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.828+0000 7f237e6b3640 1 -- 172.21.15.59:0/3818505359 shutdown_connections 2024-03-29T11:27:34.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.828+0000 7f237e6b3640 1 -- 172.21.15.59:0/3818505359 wait complete. 2024-03-29T11:27:34.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:34.828+0000 7f237e6b3640 1 librados: shutdown 2024-03-29T11:27:34.831 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:41.638 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:41.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.806+0000 7fa989f48640 10 monclient: get_monmap_and_config 2024-03-29T11:27:41.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.806+0000 7fa989f48640 10 monclient: build_initial_monmap 2024-03-29T11:27:41.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.806+0000 7fa989f48640 10 monclient: monmap: 2024-03-29T11:27:41.795 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:41.795 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:41.795 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:27:41.807675+0000 2024-03-29T11:27:41.795 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:27:41.807675+0000 2024-03-29T11:27:41.795 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:41.795 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:41.795 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:41.796 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:27:41.796 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:27:41.796 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:41.796 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.808+0000 7fa989f48640 1 Processor -- start 2024-03-29T11:27:41.796 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.808+0000 7fa989f48640 1 -- start start 2024-03-29T11:27:41.796 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.808+0000 7fa989f48640 10 monclient: init 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.808+0000 7fa989f48640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.809+0000 7fa989f48640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.809+0000 7fa989f48640 10 monclient(hunting): picked mon.noname-a con 0x7fa9841457f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.809+0000 7fa989f48640 10 monclient(hunting): picked mon.noname-c con 0x7fa98413f650 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.809+0000 7fa989f48640 10 monclient(hunting): picked mon.noname-b con 0x7fa9840a9d70 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.809+0000 7fa989f48640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9840aaef0 con 0x7fa9841457f0 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.809+0000 7fa989f48640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa984145bd0 con 0x7fa98413f650 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.809+0000 7fa989f48640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa984145eb0 con 0x7fa9840a9d70 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.809+0000 7fa989f48640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.809+0000 7fa989f48640 10 monclient(hunting): authenticate will time out at 1311.922775s 2024-03-29T11:27:41.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.813+0000 7fa983fff640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fa9840a9d70 0x7fa98414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:57485/0 (socket says 172.21.15.59:57485) 2024-03-29T11:27:41.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.814+0000 7fa983fff640 1 -- 172.21.15.59:0/2299452248 learned_addr learned my addr 172.21.15.59:0/2299452248 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:41.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.814+0000 7fa982ffd640 1 -- 172.21.15.59:0/2299452248 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891537725 0 0) 0x7fa984145eb0 con 0x7fa9840a9d70 2024-03-29T11:27:41.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.814+0000 7fa982ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.814+0000 7fa982ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.814+0000 7fa982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.814+0000 7fa982ffd640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:27:41.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.814+0000 7fa982ffd640 1 -- 172.21.15.59:0/2299452248 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa970003520 con 0x7fa9840a9d70 2024-03-29T11:27:41.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.814+0000 7fa982ffd640 1 -- 172.21.15.59:0/2299452248 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2002516136 0 0) 0x7fa970003520 con 0x7fa9840a9d70 2024-03-29T11:27:41.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.814+0000 7fa982ffd640 1 -- 172.21.15.59:0/2299452248 >> v1:172.21.15.172:6789/0 conn(0x7fa98413f650 legacy=0x7fa9840b2620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:27:41.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.814+0000 7fa982ffd640 1 -- 172.21.15.59:0/2299452248 >> v1:172.21.15.59:6789/0 conn(0x7fa9841457f0 legacy=0x7fa98413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:41.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.815+0000 7fa982ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:41.804 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.815+0000 7fa982ffd640 1 monclient: found mon.noname-b 2024-03-29T11:27:41.804 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.815+0000 7fa982ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:41.804 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.815+0000 7fa982ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:27:41.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.817+0000 7fa982ffd640 1 -- 172.21.15.59:0/2299452248 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa984146340 con 0x7fa9840a9d70 2024-03-29T11:27:41.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.817+0000 7fa982ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:41.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.817+0000 7fa982ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:41.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.817+0000 7fa982ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:41.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.817+0000 7fa982ffd640 1 -- 172.21.15.59:0/2299452248 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa97c001a10 con 0x7fa9840a9d70 2024-03-29T11:27:41.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.817+0000 7fa982ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.817+0000 7fa982ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:41.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.817+0000 7fa982ffd640 10 monclient: dump: 2024-03-29T11:27:41.806 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:41.806 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:41.807 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.807 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.807 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.807 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:41.807 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:41.807 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:41.807 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:41.807 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:41.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.817+0000 7fa989f48640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:27:41.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.817+0000 7fa989f48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:41.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.818+0000 7fa982ffd640 1 -- 172.21.15.59:0/2299452248 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa97c001be0 con 0x7fa9840a9d70 2024-03-29T11:27:41.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.818+0000 7fa982ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:41.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.818+0000 7fa982ffd640 1 -- 172.21.15.59:0/2299452248 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa97c002030 con 0x7fa9840a9d70 2024-03-29T11:27:41.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.818+0000 7fa989f48640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:41.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.818+0000 7fa989f48640 4 set_mon_vals no callback set 2024-03-29T11:27:41.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.819+0000 7fa982ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.819+0000 7fa982ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:41.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.819+0000 7fa982ffd640 10 monclient: dump: 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.819+0000 7fa989f48640 10 monclient: shutdown 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.819+0000 7fa989f48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.819+0000 7fa989f48640 1 -- 172.21.15.59:0/2299452248 >> v1:172.21.15.59:6790/0 conn(0x7fa9840a9d70 legacy=0x7fa98414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.819+0000 7fa989f48640 1 -- 172.21.15.59:0/2299452248 shutdown_connections 2024-03-29T11:27:41.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.819+0000 7fa989f48640 1 -- 172.21.15.59:0/2299452248 >> 172.21.15.59:0/2299452248 conn(0x7fa984066a00 msgr2=0x7fa984066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:41.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.819+0000 7fa989f48640 1 -- 172.21.15.59:0/2299452248 shutdown_connections 2024-03-29T11:27:41.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.819+0000 7fa989f48640 1 -- 172.21.15.59:0/2299452248 wait complete. 2024-03-29T11:27:41.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.821+0000 7fa989f48640 10 monclient: build_initial_monmap 2024-03-29T11:27:41.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.821+0000 7fa989f48640 10 monclient: monmap: 2024-03-29T11:27:41.812 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:41.812 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:41.812 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:27:41.812 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:27:41.812 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.821+0000 7fa989f48640 1 librados: starting msgr at 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.821+0000 7fa989f48640 1 librados: starting objecter 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.821+0000 7fa989f48640 1 Processor -- start 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.822+0000 7fa989f48640 1 -- start start 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.822+0000 7fa989f48640 1 librados: setting wanted keys 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.822+0000 7fa989f48640 1 librados: calling monclient init 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.822+0000 7fa989f48640 10 monclient: init 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.823+0000 7fa989f48640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.823+0000 7fa989f48640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:41.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.823+0000 7fa989f48640 10 monclient(hunting): picked mon.noname-a con 0x7fa9840a9d70 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:41.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.823+0000 7fa989f48640 10 monclient(hunting): picked mon.noname-b con 0x7fa98413f650 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:41.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.823+0000 7fa989f48640 10 monclient(hunting): picked mon.noname-c con 0x7fa9841457f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:41.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.823+0000 7fa989f48640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9841422d0 con 0x7fa9840a9d70 2024-03-29T11:27:41.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.823+0000 7fa989f48640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa984142590 con 0x7fa98413f650 2024-03-29T11:27:41.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.823+0000 7fa989f48640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9841d7620 con 0x7fa9841457f0 2024-03-29T11:27:41.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.823+0000 7fa989f48640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:41.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.823+0000 7fa989f48640 10 monclient(hunting): authenticate will time out at 1311.937020s 2024-03-29T11:27:41.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.826+0000 7fa983fff640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fa9841457f0 0x7fa9841d42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:57569/0 (socket says 172.21.15.59:57569) 2024-03-29T11:27:41.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.826+0000 7fa983fff640 1 -- 172.21.15.59:0/2980822447 learned_addr learned my addr 172.21.15.59:0/2980822447 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:41.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.829+0000 7fa980ff9640 10 client.?.objecter ms_handle_connect 0x7fa9841457f0 2024-03-29T11:27:41.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.829+0000 7fa980ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:41.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.829+0000 7fa980ff9640 10 client.?.objecter ms_handle_connect 0x7fa9840a9d70 2024-03-29T11:27:41.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.829+0000 7fa980ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:41.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.829+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429532889 0 0) 0x7fa9841422d0 con 0x7fa9840a9d70 2024-03-29T11:27:41.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.829+0000 7fa980ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.829+0000 7fa980ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.829+0000 7fa980ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.829+0000 7fa980ff9640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.829+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa964004eb0 con 0x7fa9840a9d70 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 10 client.?.objecter ms_handle_connect 0x7fa98413f650 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183802645 0 0) 0x7fa9841d7620 con 0x7fa9841457f0 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa964006230 con 0x7fa9841457f0 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602065457 0 0) 0x7fa984142590 con 0x7fa98413f650 2024-03-29T11:27:41.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.830+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9841d7620 con 0x7fa98413f650 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 897574481 0 0) 0x7fa964004eb0 con 0x7fa9840a9d70 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa984142590 con 0x7fa9840a9d70 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 254095924 0 0) 0x7fa964006230 con 0x7fa9841457f0 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa964004eb0 con 0x7fa9841457f0 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa9740016c0 con 0x7fa9840a9d70 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:41.824 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa97c001ef0 con 0x7fa9841457f0 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.831+0000 7fa980ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:41.825 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.832+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053429861 0 0) 0x7fa9841d7620 con 0x7fa98413f650 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.832+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa964006230 con 0x7fa98413f650 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.832+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa96c001730 con 0x7fa98413f650 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.832+0000 7fa980ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.832+0000 7fa980ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:41.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.832+0000 7fa980ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1290420523 0 0) 0x7fa964006230 con 0x7fa98413f650 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 >> v1:172.21.15.59:6790/0 conn(0x7fa9841457f0 legacy=0x7fa9841d42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 >> v1:172.21.15.59:6789/0 conn(0x7fa9840a9d70 legacy=0x7fa984140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa980ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa980ff9640 1 monclient: found mon.b 2024-03-29T11:27:41.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa980ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa980ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa984140d50 con 0x7fa98413f650 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa980ff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa980ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa980ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa989f48640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa989f48640 10 monclient: _renew_subs 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa989f48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa989f48640 1 -- 172.21.15.59:0/2980822447 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa984141eb0 con 0x7fa98413f650 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa989f48640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa989f48640 10 monclient: _renew_subs 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa989f48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa989f48640 1 -- 172.21.15.59:0/2980822447 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa984142a20 con 0x7fa98413f650 2024-03-29T11:27:41.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.833+0000 7fa989f48640 1 librados: init done 2024-03-29T11:27:41.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.834+0000 7fa989f48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:41.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.835+0000 7fa989f48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:41.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.835+0000 7fa989f48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:27:41.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.835+0000 7fa989f48640 1 -- 172.21.15.59:0/2980822447 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa984075f10 con 0x7fa98413f650 2024-03-29T11:27:41.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.835+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa96c001c40 con 0x7fa98413f650 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.835+0000 7fa980ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.835+0000 7fa981ffb640 4 set_mon_vals no callback set 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.835+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fa96c001730 con 0x7fa98413f650 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.835+0000 7fa980ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.835+0000 7fa980ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.835+0000 7fa980ff9640 10 monclient: dump: 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:41.831 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:41.832 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:41.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.840+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fa96c019500 con 0x7fa98413f650 2024-03-29T11:27:41.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.841+0000 7fa9837fe640 10 monclient: get_auth_request con 0x7fa96406e7f0 auth_method 0 2024-03-29T11:27:41.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.841+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fa96c001550 con 0x7fa98413f650 2024-03-29T11:27:41.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.841+0000 7fa980ff9640 10 client.4513.objecter ms_dispatch 0x7fa984000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:41.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.841+0000 7fa980ff9640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:41.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.841+0000 7fa980ff9640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:41.832 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.841+0000 7fa980ff9640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-29T11:27:41.834 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.846+0000 7fa980ff9640 10 client.4513.objecter ms_handle_connect 0x7fa96406e7f0 2024-03-29T11:27:41.836 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.847+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa96c04f250 con 0x7fa98413f650 2024-03-29T11:27:41.836 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.848+0000 7fa980ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:41.836 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:41.848+0000 7fa980ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:42.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.179+0000 7fa989f48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:42.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.179+0000 7fa989f48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:42.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.179+0000 7fa989f48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:27:42.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.179+0000 7fa989f48640 1 -- 172.21.15.59:0/2980822447 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa984142740 con 0x7fa98413f650 2024-03-29T11:27:42.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.180+0000 7fa980ff9640 1 -- 172.21.15.59:0/2980822447 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fa96c04f250 con 0x7fa98413f650 2024-03-29T11:27:42.169 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.180+0000 7fa980ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:42.169 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.180+0000 7fa980ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:42.171 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:27:42.172 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:42.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.187+0000 7fa989f48640 10 librados: watch_flush enter 2024-03-29T11:27:42.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.187+0000 7fa989f48640 10 librados: watch_flush exit 2024-03-29T11:27:42.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.187+0000 7fa989f48640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-29T11:27:42.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.187+0000 7fa989f48640 10 client.4513.objecter successfully canceled tick 2024-03-29T11:27:42.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.187+0000 7fa989f48640 1 -- 172.21.15.59:0/2980822447 >> v1:172.21.15.172:6816/2420742773 conn(0x7fa96406e7f0 legacy=0x7fa964070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.187+0000 7fa989f48640 10 monclient: shutdown 2024-03-29T11:27:42.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.187+0000 7fa989f48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:42.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.187+0000 7fa989f48640 1 -- 172.21.15.59:0/2980822447 >> v1:172.21.15.172:6789/0 conn(0x7fa98413f650 legacy=0x7fa9841d6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.188+0000 7fa989f48640 1 -- 172.21.15.59:0/2980822447 shutdown_connections 2024-03-29T11:27:42.177 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.188+0000 7fa989f48640 1 -- 172.21.15.59:0/2980822447 >> 172.21.15.59:0/2980822447 conn(0x7fa984066a00 msgr2=0x7fa9840b4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:42.177 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.188+0000 7fa989f48640 1 -- 172.21.15.59:0/2980822447 shutdown_connections 2024-03-29T11:27:42.177 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.188+0000 7fa989f48640 1 -- 172.21.15.59:0/2980822447 wait complete. 2024-03-29T11:27:42.177 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:42.188+0000 7fa989f48640 1 librados: shutdown 2024-03-29T11:27:42.196 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:49.004 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:49.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.164+0000 7f5b965ec640 10 monclient: get_monmap_and_config 2024-03-29T11:27:49.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.164+0000 7f5b965ec640 10 monclient: build_initial_monmap 2024-03-29T11:27:49.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.164+0000 7f5b965ec640 10 monclient: monmap: 2024-03-29T11:27:49.153 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:49.153 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:49.153 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:27:49.165079+0000 2024-03-29T11:27:49.153 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:27:49.165079+0000 2024-03-29T11:27:49.154 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:49.154 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:49.154 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:49.154 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:27:49.154 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:27:49.154 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:49.156 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.167+0000 7f5b965ec640 1 Processor -- start 2024-03-29T11:27:49.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.169+0000 7f5b965ec640 1 -- start start 2024-03-29T11:27:49.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.169+0000 7f5b965ec640 10 monclient: init 2024-03-29T11:27:49.159 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.171+0000 7f5b965ec640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:49.159 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.171+0000 7f5b965ec640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:49.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.171+0000 7f5b965ec640 10 monclient(hunting): picked mon.noname-a con 0x7f5b88006a70 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:49.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.171+0000 7f5b965ec640 10 monclient(hunting): picked mon.noname-b con 0x7f5b8802b100 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:49.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.171+0000 7f5b965ec640 10 monclient(hunting): picked mon.noname-c con 0x7f5b88029c30 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:49.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.171+0000 7f5b965ec640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8802a270 con 0x7f5b88006a70 2024-03-29T11:27:49.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.171+0000 7f5b965ec640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8802c440 con 0x7f5b88029c30 2024-03-29T11:27:49.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.171+0000 7f5b965ec640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b880c9440 con 0x7f5b8802b100 2024-03-29T11:27:49.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.171+0000 7f5b965ec640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:49.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.171+0000 7f5b965ec640 10 monclient(hunting): authenticate will time out at 1319.285219s 2024-03-29T11:27:49.165 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.176+0000 7f5b95deb640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5b88029c30 0x7f5b880d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:54991/0 (socket says 172.21.15.59:54991) 2024-03-29T11:27:49.165 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.176+0000 7f5b95deb640 1 -- 172.21.15.59:0/908749037 learned_addr learned my addr 172.21.15.59:0/908749037 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:49.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.179+0000 7f5b87fff640 1 -- 172.21.15.59:0/908749037 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794724032 0 0) 0x7f5b8802a270 con 0x7f5b88006a70 2024-03-29T11:27:49.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.179+0000 7f5b87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:49.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.179+0000 7f5b87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:49.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.179+0000 7f5b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:49.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.179+0000 7f5b87fff640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:27:49.168 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.179+0000 7f5b87fff640 1 -- 172.21.15.59:0/908749037 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7c003520 con 0x7f5b88006a70 2024-03-29T11:27:49.172 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.179+0000 7f5b87fff640 1 -- 172.21.15.59:0/908749037 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 722199760 0 0) 0x7f5b7c003520 con 0x7f5b88006a70 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.179+0000 7f5b87fff640 1 -- 172.21.15.59:0/908749037 >> v1:172.21.15.59:6790/0 conn(0x7f5b8802b100 legacy=0x7f5b880cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 1 -- 172.21.15.59:0/908749037 >> v1:172.21.15.172:6789/0 conn(0x7f5b88029c30 legacy=0x7f5b880d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 1 monclient: found mon.noname-a 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 1 -- 172.21.15.59:0/908749037 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b880c95d0 con 0x7f5b88006a70 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 1 -- 172.21.15.59:0/908749037 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5b8c001690 con 0x7f5b88006a70 2024-03-29T11:27:49.173 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.174 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:49.174 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.180+0000 7f5b87fff640 10 monclient: dump: 2024-03-29T11:27:49.174 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:49.174 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:49.174 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:49.174 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:49.174 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.174 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:49.174 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:49.174 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:49.175 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:49.175 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:49.175 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.181+0000 7f5b87fff640 1 -- 172.21.15.59:0/908749037 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b8c001860 con 0x7f5b88006a70 2024-03-29T11:27:49.175 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.181+0000 7f5b87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:49.175 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.181+0000 7f5b87fff640 1 -- 172.21.15.59:0/908749037 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5b8c001c70 con 0x7f5b88006a70 2024-03-29T11:27:49.175 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.181+0000 7f5b87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.175 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.181+0000 7f5b87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:49.175 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.181+0000 7f5b87fff640 10 monclient: dump: 2024-03-29T11:27:49.175 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:49.175 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:49.175 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.182+0000 7f5b965ec640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.182+0000 7f5b965ec640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.182+0000 7f5b965ec640 4 set_mon_vals no callback set 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.183+0000 7f5b965ec640 10 monclient: shutdown 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.183+0000 7f5b965ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.183+0000 7f5b965ec640 1 -- 172.21.15.59:0/908749037 >> v1:172.21.15.59:6789/0 conn(0x7f5b88006a70 legacy=0x7f5b880c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.184+0000 7f5b965ec640 1 -- 172.21.15.59:0/908749037 shutdown_connections 2024-03-29T11:27:49.177 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.184+0000 7f5b965ec640 1 -- 172.21.15.59:0/908749037 >> 172.21.15.59:0/908749037 conn(0x7f5b880058f0 msgr2=0x7f5b88005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:49.177 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.184+0000 7f5b965ec640 1 -- 172.21.15.59:0/908749037 shutdown_connections 2024-03-29T11:27:49.177 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.184+0000 7f5b965ec640 1 -- 172.21.15.59:0/908749037 wait complete. 2024-03-29T11:27:49.178 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.185+0000 7f5b965ec640 10 monclient: build_initial_monmap 2024-03-29T11:27:49.178 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.185+0000 7f5b965ec640 10 monclient: monmap: 2024-03-29T11:27:49.178 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:49.178 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:49.178 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:27:49.178 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:27:49.178 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:49.178 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:49.178 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.186+0000 7f5b965ec640 1 librados: starting msgr at 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.186+0000 7f5b965ec640 1 librados: starting objecter 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.186+0000 7f5b965ec640 1 Processor -- start 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.186+0000 7f5b965ec640 1 -- start start 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.186+0000 7f5b965ec640 1 librados: setting wanted keys 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.186+0000 7f5b965ec640 1 librados: calling monclient init 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.186+0000 7f5b965ec640 10 monclient: init 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.187+0000 7f5b965ec640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:49.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.187+0000 7f5b965ec640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.187+0000 7f5b965ec640 10 monclient(hunting): picked mon.noname-c con 0x7f5b88006a70 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.187+0000 7f5b965ec640 10 monclient(hunting): picked mon.noname-b con 0x7f5b88029c30 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.188+0000 7f5b965ec640 10 monclient(hunting): picked mon.noname-a con 0x7f5b8802e0b0 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.188+0000 7f5b965ec640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8802f530 con 0x7f5b8802e0b0 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.188+0000 7f5b965ec640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8802d480 con 0x7f5b88029c30 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.188+0000 7f5b965ec640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b880ca890 con 0x7f5b88006a70 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.188+0000 7f5b965ec640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.188+0000 7f5b965ec640 10 monclient(hunting): authenticate will time out at 1319.302501s 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.189+0000 7f5b955ea640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f5b88006a70 0x7f5b8802d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:39125/0 (socket says 172.21.15.59:39125) 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.189+0000 7f5b955ea640 1 -- 172.21.15.59:0/2927405133 learned_addr learned my addr 172.21.15.59:0/2927405133 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.189+0000 7f5b85ffb640 10 client.?.objecter ms_handle_connect 0x7f5b88006a70 2024-03-29T11:27:49.180 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.189+0000 7f5b85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:49.184 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.190+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640073928 0 0) 0x7f5b880ca890 con 0x7f5b88006a70 2024-03-29T11:27:49.184 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.190+0000 7f5b85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:49.184 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.190+0000 7f5b85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:49.184 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.190+0000 7f5b85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:49.184 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.190+0000 7f5b85ffb640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:27:49.184 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.190+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b74004eb0 con 0x7f5b88006a70 2024-03-29T11:27:49.184 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 372391333 0 0) 0x7f5b74004eb0 con 0x7f5b88006a70 2024-03-29T11:27:49.184 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b880ca890 con 0x7f5b88006a70 2024-03-29T11:27:49.184 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5b8c001c30 con 0x7f5b88006a70 2024-03-29T11:27:49.184 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.185 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:49.185 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:49.185 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:49.185 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:49.185 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:49.185 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:49.185 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.185 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:49.185 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 10 client.?.objecter ms_handle_connect 0x7f5b88029c30 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2646337012 0 0) 0x7f5b880ca890 con 0x7f5b88006a70 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 >> v1:172.21.15.172:6789/0 conn(0x7f5b88029c30 legacy=0x7f5b88031d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.191+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 >> v1:172.21.15.59:6789/0 conn(0x7f5b8802e0b0 legacy=0x7f5b8816c430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.193+0000 7f5b85ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.193+0000 7f5b85ffb640 1 monclient: found mon.c 2024-03-29T11:27:49.186 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.193+0000 7f5b85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.193+0000 7f5b85ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.193+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b880caa20 con 0x7f5b88006a70 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.193+0000 7f5b85ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.193+0000 7f5b85ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.193+0000 7f5b85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.193+0000 7f5b965ec640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.194+0000 7f5b965ec640 10 monclient: _renew_subs 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.194+0000 7f5b965ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.194+0000 7f5b965ec640 1 -- 172.21.15.59:0/2927405133 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b88164d20 con 0x7f5b88006a70 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.194+0000 7f5b965ec640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.194+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b8c003940 con 0x7f5b88006a70 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.194+0000 7f5b85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:49.187 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.194+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5b8c003bb0 con 0x7f5b88006a70 2024-03-29T11:27:49.188 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.194+0000 7f5b965ec640 10 monclient: _renew_subs 2024-03-29T11:27:49.188 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.194+0000 7f5b965ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:49.188 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.194+0000 7f5b965ec640 1 -- 172.21.15.59:0/2927405133 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b88032670 con 0x7f5b88006a70 2024-03-29T11:27:49.188 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.195+0000 7f5b965ec640 1 librados: init done 2024-03-29T11:27:49.188 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.195+0000 7f5b85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:49.189 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.196+0000 7f5b85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:49.190 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.196+0000 7f5b85ffb640 10 monclient: dump: 2024-03-29T11:27:49.190 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:49.190 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:49.190 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:49.190 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:49.190 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:49.190 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:49.190 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:49.190 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:49.190 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:49.190 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.196+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f5b8c003d10 con 0x7f5b88006a70 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.196+0000 7f5b86ffd640 4 set_mon_vals no callback set 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.196+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f5b8c084bf0 con 0x7f5b88006a70 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.196+0000 7f5b85ffb640 10 client.4541.objecter ms_dispatch 0x7f5b90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.196+0000 7f5b85ffb640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.196+0000 7f5b85ffb640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.196+0000 7f5b85ffb640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.197+0000 7f5b94de9640 10 monclient: get_auth_request con 0x7f5b7406dbe0 auth_method 0 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.198+0000 7f5b965ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.198+0000 7f5b965ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.198+0000 7f5b965ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:49.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.198+0000 7f5b965ec640 1 -- 172.21.15.59:0/2927405133 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b70005080 con 0x7f5b88006a70 2024-03-29T11:27:49.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.200+0000 7f5b85ffb640 10 client.4541.objecter ms_handle_connect 0x7f5b7406dbe0 2024-03-29T11:27:49.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.210+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b8c0504f0 con 0x7f5b88006a70 2024-03-29T11:27:49.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.210+0000 7f5b85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:49.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.210+0000 7f5b85ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:49.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.549+0000 7f5b965ec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.549+0000 7f5b965ec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.549+0000 7f5b965ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:49.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.549+0000 7f5b965ec640 1 -- 172.21.15.59:0/2927405133 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b70005330 con 0x7f5b88006a70 2024-03-29T11:27:49.540 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.551+0000 7f5b85ffb640 1 -- 172.21.15.59:0/2927405133 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f5b8c052420 con 0x7f5b88006a70 2024-03-29T11:27:49.540 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.551+0000 7f5b85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.540 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.551+0000 7f5b85ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:49.543 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:27:49.543 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:49.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.555+0000 7f5b7b7fe640 10 librados: watch_flush enter 2024-03-29T11:27:49.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.555+0000 7f5b7b7fe640 10 librados: watch_flush exit 2024-03-29T11:27:49.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.555+0000 7f5b7b7fe640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T11:27:49.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.555+0000 7f5b7b7fe640 10 client.4541.objecter successfully canceled tick 2024-03-29T11:27:49.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.555+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2927405133 >> v1:172.21.15.172:6816/2420742773 conn(0x7f5b7406dbe0 legacy=0x7f5b740700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.545 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.555+0000 7f5b7b7fe640 10 monclient: shutdown 2024-03-29T11:27:49.546 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.555+0000 7f5b7b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:49.546 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.555+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2927405133 >> v1:172.21.15.59:6790/0 conn(0x7f5b88006a70 legacy=0x7f5b8802d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.546 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.555+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2927405133 shutdown_connections 2024-03-29T11:27:49.546 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.555+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2927405133 >> 172.21.15.59:0/2927405133 conn(0x7f5b880058f0 msgr2=0x7f5b880c73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:49.546 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.556+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2927405133 shutdown_connections 2024-03-29T11:27:49.546 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.556+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2927405133 wait complete. 2024-03-29T11:27:49.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:49.557+0000 7f5b7b7fe640 1 librados: shutdown 2024-03-29T11:27:49.566 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:56.374 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:56.528 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.539+0000 7f4ab13af640 10 monclient: get_monmap_and_config 2024-03-29T11:27:56.528 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.539+0000 7f4ab13af640 10 monclient: build_initial_monmap 2024-03-29T11:27:56.528 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.539+0000 7f4ab13af640 10 monclient: monmap: 2024-03-29T11:27:56.528 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:56.529 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:56.529 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:27:56.540017+0000 2024-03-29T11:27:56.529 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:27:56.540017+0000 2024-03-29T11:27:56.529 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:56.529 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:56.529 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:56.529 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:27:56.530 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:27:56.530 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:56.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.540+0000 7f4ab13af640 1 Processor -- start 2024-03-29T11:27:56.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.541+0000 7f4ab13af640 1 -- start start 2024-03-29T11:27:56.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.541+0000 7f4ab13af640 10 monclient: init 2024-03-29T11:27:56.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.542+0000 7f4ab13af640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:56.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.542+0000 7f4ab13af640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:56.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.542+0000 7f4ab13af640 10 monclient(hunting): picked mon.noname-c con 0x7f4aac0b0860 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:56.534 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.542+0000 7f4ab13af640 10 monclient(hunting): picked mon.noname-b con 0x7f4aac072450 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:56.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.548+0000 7f4aaaffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4aac0b0860 0x7f4aac0acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:48015/0 (socket says 172.21.15.59:48015) 2024-03-29T11:27:56.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.548+0000 7f4aaaffd640 1 -- 172.21.15.59:0/99614261 learned_addr learned my addr 172.21.15.59:0/99614261 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:56.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.548+0000 7f4ab13af640 10 monclient(hunting): picked mon.noname-a con 0x7f4aac13ee40 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:56.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.548+0000 7f4ab13af640 1 -- 172.21.15.59:0/99614261 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0ad840 con 0x7f4aac13ee40 2024-03-29T11:27:56.538 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.548+0000 7f4ab13af640 1 -- 172.21.15.59:0/99614261 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac13c830 con 0x7f4aac0b0860 2024-03-29T11:27:56.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.548+0000 7f4ab13af640 1 -- 172.21.15.59:0/99614261 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac13c9c0 con 0x7f4aac072450 2024-03-29T11:27:56.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.548+0000 7f4ab13af640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:56.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.548+0000 7f4ab13af640 10 monclient(hunting): authenticate will time out at 1326.662543s 2024-03-29T11:27:56.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.550+0000 7f4aa9ffb640 1 -- 172.21.15.59:0/99614261 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414691405 0 0) 0x7f4aac0ad840 con 0x7f4aac13ee40 2024-03-29T11:27:56.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.550+0000 7f4aa9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.550+0000 7f4aa9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.550+0000 7f4aa9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.550+0000 7f4aa9ffb640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:27:56.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.550+0000 7f4aa9ffb640 1 -- 172.21.15.59:0/99614261 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a94003520 con 0x7f4aac13ee40 2024-03-29T11:27:56.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.550+0000 7f4aa9ffb640 1 -- 172.21.15.59:0/99614261 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108761249 0 0) 0x7f4aac13c9c0 con 0x7f4aac072450 2024-03-29T11:27:56.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.550+0000 7f4aa9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.550+0000 7f4aa9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.549 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 1 -- 172.21.15.59:0/99614261 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aac0ad840 con 0x7f4aac072450 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 1 -- 172.21.15.59:0/99614261 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 300936340 0 0) 0x7f4a94003520 con 0x7f4aac13ee40 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 1 -- 172.21.15.59:0/99614261 >> v1:172.21.15.59:6790/0 conn(0x7f4aac072450 legacy=0x7f4aac13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 1 -- 172.21.15.59:0/99614261 >> v1:172.21.15.172:6789/0 conn(0x7f4aac0b0860 legacy=0x7f4aac0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 1 monclient: found mon.noname-a 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 1 -- 172.21.15.59:0/99614261 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac13f970 con 0x7f4aac13ee40 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:56.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 1 -- 172.21.15.59:0/99614261 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4aa4001a10 con 0x7f4aac13ee40 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.551+0000 7f4aa9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.552+0000 7f4aa9ffb640 10 monclient: dump: 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:56.551 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.552+0000 7f4aa9ffb640 1 -- 172.21.15.59:0/99614261 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aa4001c30 con 0x7f4aac13ee40 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.552+0000 7f4aa9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.552+0000 7f4aa9ffb640 1 -- 172.21.15.59:0/99614261 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4aa4001fc0 con 0x7f4aac13ee40 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.552+0000 7f4aa9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.552+0000 7f4aa9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.552+0000 7f4aa9ffb640 10 monclient: dump: 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:56.552 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.552+0000 7f4ab13af640 5 monclient: authenticate success, global_id 4527 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.552+0000 7f4ab13af640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.552+0000 7f4ab13af640 4 set_mon_vals no callback set 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.553+0000 7f4ab13af640 10 monclient: shutdown 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.553+0000 7f4ab13af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.553+0000 7f4ab13af640 1 -- 172.21.15.59:0/99614261 >> v1:172.21.15.59:6789/0 conn(0x7f4aac13ee40 legacy=0x7f4aac13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.554+0000 7f4ab13af640 1 -- 172.21.15.59:0/99614261 shutdown_connections 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.554+0000 7f4ab13af640 1 -- 172.21.15.59:0/99614261 >> 172.21.15.59:0/99614261 conn(0x7f4aac06b660 msgr2=0x7f4aac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.554+0000 7f4ab13af640 1 -- 172.21.15.59:0/99614261 shutdown_connections 2024-03-29T11:27:56.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.554+0000 7f4ab13af640 1 -- 172.21.15.59:0/99614261 wait complete. 2024-03-29T11:27:56.554 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.555+0000 7f4ab13af640 10 monclient: build_initial_monmap 2024-03-29T11:27:56.554 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.555+0000 7f4ab13af640 10 monclient: monmap: 2024-03-29T11:27:56.554 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:27:56.554 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:56.554 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:27:56.554 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:27:56.554 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:56.554 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:56.554 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:27:56.554 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:27:56.554 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:27:56.555 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:56.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.556+0000 7f4ab13af640 1 librados: starting msgr at 2024-03-29T11:27:56.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.556+0000 7f4ab13af640 1 librados: starting objecter 2024-03-29T11:27:56.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.556+0000 7f4ab13af640 1 Processor -- start 2024-03-29T11:27:56.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.556+0000 7f4ab13af640 1 -- start start 2024-03-29T11:27:56.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.557+0000 7f4ab13af640 1 librados: setting wanted keys 2024-03-29T11:27:56.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.557+0000 7f4ab13af640 1 librados: calling monclient init 2024-03-29T11:27:56.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.557+0000 7f4ab13af640 10 monclient: init 2024-03-29T11:27:56.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.558+0000 7f4ab13af640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:56.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.558+0000 7f4ab13af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:56.557 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.561+0000 7f4aaaffd640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f4aac072450 0x7f4aac06e3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:41651/0 (socket says 172.21.15.59:41651) 2024-03-29T11:27:56.557 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.561+0000 7f4aaaffd640 1 -- 172.21.15.59:0/476895798 learned_addr learned my addr 172.21.15.59:0/476895798 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.561+0000 7f4ab13af640 10 monclient(hunting): picked mon.noname-c con 0x7f4aac072450 addr v1:172.21.15.59:6790/0 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.561+0000 7f4a937fe640 10 client.?.objecter ms_handle_connect 0x7f4aac072450 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.563+0000 7f4a937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.561+0000 7f4ab13af640 10 monclient(hunting): picked mon.noname-a con 0x7f4aac0b0860 addr v1:172.21.15.59:6789/0 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.563+0000 7f4ab13af640 10 monclient(hunting): picked mon.noname-b con 0x7f4aac13ee40 addr v1:172.21.15.172:6789/0 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.563+0000 7f4ab13af640 1 -- 172.21.15.59:0/476895798 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0710c0 con 0x7f4aac0b0860 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.564+0000 7f4ab13af640 1 -- 172.21.15.59:0/476895798 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0713a0 con 0x7f4aac13ee40 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.564+0000 7f4ab13af640 1 -- 172.21.15.59:0/476895798 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac071680 con 0x7f4aac072450 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.564+0000 7f4ab13af640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.564+0000 7f4ab13af640 10 monclient(hunting): authenticate will time out at 1326.677849s 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.564+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850162600 0 0) 0x7f4aac071680 con 0x7f4aac072450 2024-03-29T11:27:56.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.564+0000 7f4a937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.564+0000 7f4a937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.564+0000 7f4a937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.564+0000 7f4a937fe640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.564+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a98004eb0 con 0x7f4aac072450 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.566+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3289109045 0 0) 0x7f4a98004eb0 con 0x7f4aac072450 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.566+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4aac071680 con 0x7f4aac072450 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.566+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4aa00018b0 con 0x7f4aac072450 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.566+0000 7f4a937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.566+0000 7f4a937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.566+0000 7f4a937fe640 10 monclient(hunting): dump: 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:56.559 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:56.560 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:56.560 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:56.560 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.560 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:56.560 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:56.560 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:56.560 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:56.560 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:56.560 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.566+0000 7f4a937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:56.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.569+0000 7f4a937fe640 10 client.?.objecter ms_handle_connect 0x7f4aac0b0860 2024-03-29T11:27:56.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.569+0000 7f4a937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:56.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.569+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769062908 0 0) 0x7f4aac0710c0 con 0x7f4aac0b0860 2024-03-29T11:27:56.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.569+0000 7f4a937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.569+0000 7f4a937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.569+0000 7f4a937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.569+0000 7f4a937fe640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:27:56.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.569+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a98004eb0 con 0x7f4aac0b0860 2024-03-29T11:27:56.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.569+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1054447122 0 0) 0x7f4aac071680 con 0x7f4aac072450 2024-03-29T11:27:56.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.569+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 >> v1:172.21.15.172:6789/0 conn(0x7f4aac13ee40 legacy=0x7f4aac12e380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:56.563 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 >> v1:172.21.15.59:6789/0 conn(0x7f4aac0b0860 legacy=0x7f4aac0b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.563 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:56.563 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 1 monclient: found mon.c 2024-03-29T11:27:56.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:56.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:56.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac06ded0 con 0x7f4aac072450 2024-03-29T11:27:56.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:56.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:56.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:56.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aa0001a10 con 0x7f4aac072450 2024-03-29T11:27:56.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:56.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4aa0004ac0 con 0x7f4aac072450 2024-03-29T11:27:56.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.564 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:56.565 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.574+0000 7f4a937fe640 10 monclient: dump: 2024-03-29T11:27:56.565 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:27:56.565 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:27:56.565 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:56.565 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:27:56.565 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.565 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:27:56.566 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:27:56.566 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:27:56.566 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:27:56.566 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:27:56.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.575+0000 7f4ab13af640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:27:56.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.575+0000 7f4ab13af640 10 monclient: _renew_subs 2024-03-29T11:27:56.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.575+0000 7f4ab13af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:56.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.575+0000 7f4ab13af640 1 -- 172.21.15.59:0/476895798 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aac06d9d0 con 0x7f4aac072450 2024-03-29T11:27:56.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.575+0000 7f4ab13af640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:56.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.575+0000 7f4ab13af640 10 monclient: _renew_subs 2024-03-29T11:27:56.567 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.575+0000 7f4ab13af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:56.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.575+0000 7f4ab13af640 1 -- 172.21.15.59:0/476895798 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aac06dc80 con 0x7f4aac072450 2024-03-29T11:27:56.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.575+0000 7f4ab13af640 1 librados: init done 2024-03-29T11:27:56.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.575+0000 7f4aa97fa640 4 set_mon_vals no callback set 2024-03-29T11:27:56.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.577+0000 7f4a917fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.577+0000 7f4a917fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.577+0000 7f4a917fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:56.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.577+0000 7f4a917fa640 1 -- 172.21.15.59:0/476895798 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a84005080 con 0x7f4aac072450 2024-03-29T11:27:56.568 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.577+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f4aa0004c20 con 0x7f4aac072450 2024-03-29T11:27:56.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.581+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f4aa0084b10 con 0x7f4aac072450 2024-03-29T11:27:56.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.582+0000 7f4a937fe640 10 client.4547.objecter ms_dispatch 0x7f4aac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:56.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.582+0000 7f4aaa7fc640 10 monclient: get_auth_request con 0x7f4a9806daf0 auth_method 0 2024-03-29T11:27:56.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.582+0000 7f4a937fe640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:56.573 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.582+0000 7f4a937fe640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:56.573 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.582+0000 7f4a937fe640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:27:56.573 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.582+0000 7f4a937fe640 10 client.4547.objecter ms_handle_connect 0x7f4a9806daf0 2024-03-29T11:27:56.576 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.587+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4aa0050220 con 0x7f4aac072450 2024-03-29T11:27:56.576 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.587+0000 7f4a937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.576 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.587+0000 7f4a937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:56.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.918+0000 7f4a917fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:56.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.918+0000 7f4a917fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:56.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.918+0000 7f4a917fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:27:56.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.918+0000 7f4a917fa640 1 -- 172.21.15.59:0/476895798 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a84005330 con 0x7f4aac072450 2024-03-29T11:27:56.909 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.921+0000 7f4a937fe640 1 -- 172.21.15.59:0/476895798 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f4aa0052320 con 0x7f4aac072450 2024-03-29T11:27:56.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.921+0000 7f4a937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:56.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.921+0000 7f4a937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:56.911 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:27:56.911 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:56.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.924+0000 7f4a917fa640 10 librados: watch_flush enter 2024-03-29T11:27:56.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.925+0000 7f4a917fa640 10 librados: watch_flush exit 2024-03-29T11:27:56.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.925+0000 7f4a917fa640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:27:56.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.925+0000 7f4a917fa640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:27:56.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.926+0000 7f4a917fa640 1 -- 172.21.15.59:0/476895798 >> v1:172.21.15.172:6816/2420742773 conn(0x7f4a9806daf0 legacy=0x7f4a9806ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.926+0000 7f4a917fa640 10 monclient: shutdown 2024-03-29T11:27:56.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.926+0000 7f4a917fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:56.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.926+0000 7f4a917fa640 1 -- 172.21.15.59:0/476895798 >> v1:172.21.15.59:6790/0 conn(0x7f4aac072450 legacy=0x7f4aac06e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.926+0000 7f4a917fa640 1 -- 172.21.15.59:0/476895798 shutdown_connections 2024-03-29T11:27:56.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.926+0000 7f4a917fa640 1 -- 172.21.15.59:0/476895798 >> 172.21.15.59:0/476895798 conn(0x7f4aac06b660 msgr2=0x7f4aac13bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:56.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.926+0000 7f4a917fa640 1 -- 172.21.15.59:0/476895798 shutdown_connections 2024-03-29T11:27:56.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.927+0000 7f4a917fa640 1 -- 172.21.15.59:0/476895798 wait complete. 2024-03-29T11:27:56.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:27:56.928+0000 7f4a917fa640 1 librados: shutdown 2024-03-29T11:27:56.936 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:03.744 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:03.900 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.911+0000 7f98d7ded640 10 monclient: get_monmap_and_config 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.911+0000 7f98d7ded640 10 monclient: build_initial_monmap 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.911+0000 7f98d7ded640 10 monclient: monmap: 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:28:03.912608+0000 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:28:03.912608+0000 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:28:03.901 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:03.902 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.912+0000 7f98d7ded640 1 Processor -- start 2024-03-29T11:28:03.902 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.913+0000 7f98d7ded640 1 -- start start 2024-03-29T11:28:03.902 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.913+0000 7f98d7ded640 10 monclient: init 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d7ded640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d7ded640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d7ded640 10 monclient(hunting): picked mon.noname-c con 0x7f98d01457f0 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d7ded640 10 monclient(hunting): picked mon.noname-a con 0x7f98d013f650 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d7ded640 10 monclient(hunting): picked mon.noname-b con 0x7f98d00a9d70 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d7ded640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00aaef0 con 0x7f98d013f650 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d7ded640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d0145bd0 con 0x7f98d01457f0 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d7ded640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d0145eb0 con 0x7f98d00a9d70 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d7ded640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d7ded640 10 monclient(hunting): authenticate will time out at 1334.028103s 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d638a640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f98d00a9d70 0x7f98d014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:55008/0 (socket says 172.21.15.59:55008) 2024-03-29T11:28:03.903 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.914+0000 7f98d638a640 1 -- 172.21.15.59:0/1830477675 learned_addr learned my addr 172.21.15.59:0/1830477675 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 1 -- 172.21.15.59:0/1830477675 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445982883 0 0) 0x7f98d0145bd0 con 0x7f98d01457f0 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 10 monclient(hunting): _negotiate 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 1 -- 172.21.15.59:0/1830477675 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc003520 con 0x7f98d01457f0 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 1 -- 172.21.15.59:0/1830477675 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 978384744 0 0) 0x7f98bc003520 con 0x7f98d01457f0 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 1 -- 172.21.15.59:0/1830477675 >> v1:172.21.15.59:6790/0 conn(0x7f98d00a9d70 legacy=0x7f98d014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 1 -- 172.21.15.59:0/1830477675 >> v1:172.21.15.59:6789/0 conn(0x7f98d013f650 legacy=0x7f98d00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 10 monclient: _finish_hunting 0 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 1 monclient: found mon.noname-c 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:28:03.905 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 1 -- 172.21.15.59:0/1830477675 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d0146340 con 0x7f98d01457f0 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 10 monclient: _finish_auth 0 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 10 monclient: _check_auth_tickets 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d7ded640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d7ded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 1 -- 172.21.15.59:0/1830477675 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f98cc001670 con 0x7f98d01457f0 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d4b87640 10 monclient: dump: 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.906 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.915+0000 7f98d7ded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d4b87640 1 -- 172.21.15.59:0/1830477675 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98cc001900 con 0x7f98d01457f0 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d4b87640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d4b87640 1 -- 172.21.15.59:0/1830477675 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f98cc001cd0 con 0x7f98d01457f0 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d4b87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d4b87640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d4b87640 10 monclient: dump: 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:03.907 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:03.908 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.908 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:03.908 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:03.908 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:03.908 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:03.908 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:03.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d7ded640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:03.908 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d7ded640 4 set_mon_vals no callback set 2024-03-29T11:28:03.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d7ded640 10 monclient: shutdown 2024-03-29T11:28:03.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d7ded640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:03.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d7ded640 1 -- 172.21.15.59:0/1830477675 >> v1:172.21.15.172:6789/0 conn(0x7f98d01457f0 legacy=0x7f98d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:03.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.916+0000 7f98d7ded640 1 -- 172.21.15.59:0/1830477675 shutdown_connections 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.917+0000 7f98d7ded640 1 -- 172.21.15.59:0/1830477675 >> 172.21.15.59:0/1830477675 conn(0x7f98d0066a00 msgr2=0x7f98d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.917+0000 7f98d7ded640 1 -- 172.21.15.59:0/1830477675 shutdown_connections 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.917+0000 7f98d7ded640 1 -- 172.21.15.59:0/1830477675 wait complete. 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.918+0000 7f98d7ded640 10 monclient: build_initial_monmap 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.918+0000 7f98d7ded640 10 monclient: monmap: 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:28:03.912 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.918+0000 7f98d7ded640 1 librados: starting msgr at 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.918+0000 7f98d7ded640 1 librados: starting objecter 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.918+0000 7f98d7ded640 1 Processor -- start 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.918+0000 7f98d7ded640 1 -- start start 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.918+0000 7f98d7ded640 1 librados: setting wanted keys 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.918+0000 7f98d7ded640 1 librados: calling monclient init 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.918+0000 7f98d7ded640 10 monclient: init 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.919+0000 7f98d7ded640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.919+0000 7f98d7ded640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.919+0000 7f98d7ded640 10 monclient(hunting): picked mon.noname-b con 0x7f98d00a9d70 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.919+0000 7f98d7ded640 10 monclient(hunting): picked mon.noname-c con 0x7f98d013f650 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98d7ded640 10 monclient(hunting): picked mon.noname-a con 0x7f98d01457f0 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98d7ded640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d006fc30 con 0x7f98d01457f0 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98d7ded640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d013ba20 con 0x7f98d00a9d70 2024-03-29T11:28:03.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98d7ded640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d006fe80 con 0x7f98d013f650 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98d7ded640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98d7ded640 10 monclient(hunting): authenticate will time out at 1334.033784s 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98d5b89640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f98d00a9d70 0x7f98d006e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:45352/0 (socket says 172.21.15.59:45352) 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98d5b89640 1 -- 172.21.15.59:0/118348871 learned_addr learned my addr 172.21.15.59:0/118348871 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98c67fc640 10 client.?.objecter ms_handle_connect 0x7f98d00a9d70 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12779677 0 0) 0x7f98d013ba20 con 0x7f98d00a9d70 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98c67fc640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.920+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b4004eb0 con 0x7f98d00a9d70 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.921+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 928546983 0 0) 0x7f98b4004eb0 con 0x7f98d00a9d70 2024-03-29T11:28:03.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.921+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d013ba20 con 0x7f98d00a9d70 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.921+0000 7f98c67fc640 10 client.?.objecter ms_handle_connect 0x7f98d013f650 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.921+0000 7f98c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.921+0000 7f98c67fc640 10 client.?.objecter ms_handle_connect 0x7f98d01457f0 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.921+0000 7f98c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.921+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f98cc001670 con 0x7f98d00a9d70 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.921+0000 7f98c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.921+0000 7f98c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.921+0000 7f98c67fc640 10 monclient(hunting): dump: 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:03.915 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.921+0000 7f98c67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2267446735 0 0) 0x7f98d013ba20 con 0x7f98d00a9d70 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 >> v1:172.21.15.59:6790/0 conn(0x7f98d013f650 legacy=0x7f98d00716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 >> v1:172.21.15.59:6789/0 conn(0x7f98d01457f0 legacy=0x7f98d0138fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 1 monclient: found mon.b 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d006ec10 con 0x7f98d00a9d70 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:03.916 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98d7ded640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98d7ded640 10 monclient: _renew_subs 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98d7ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98d7ded640 1 -- 172.21.15.59:0/118348871 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d006e810 con 0x7f98d00a9d70 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98d7ded640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98d7ded640 10 monclient: _renew_subs 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98d7ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98d7ded640 1 -- 172.21.15.59:0/118348871 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d0071fd0 con 0x7f98d00a9d70 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98d7ded640 1 librados: init done 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98cc0038a0 con 0x7f98d00a9d70 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f98cc001670 con 0x7f98d00a9d70 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c77fe640 4 set_mon_vals no callback set 2024-03-29T11:28:03.917 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.922+0000 7f98c67fc640 10 monclient: dump: 2024-03-29T11:28:03.918 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:03.918 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:03.918 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:03.918 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:03.918 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.918 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:03.918 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:03.918 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:03.918 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:03.918 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:03.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.924+0000 7f98d7ded640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:03.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.924+0000 7f98d7ded640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:03.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.925+0000 7f98d7ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:03.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.925+0000 7f98d7ded640 1 -- 172.21.15.59:0/118348871 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b8005080 con 0x7f98d00a9d70 2024-03-29T11:28:03.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.926+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f98cc003ad0 con 0x7f98d00a9d70 2024-03-29T11:28:03.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.926+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f98cc084970 con 0x7f98d00a9d70 2024-03-29T11:28:03.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.926+0000 7f98c67fc640 10 client.4522.objecter ms_dispatch 0x7f98d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:03.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.926+0000 7f98c67fc640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:03.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.926+0000 7f98c67fc640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:03.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.926+0000 7f98c67fc640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-29T11:28:03.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.927+0000 7f98d5388640 10 monclient: get_auth_request con 0x7f98b406de60 auth_method 0 2024-03-29T11:28:03.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.927+0000 7f98c67fc640 10 client.4522.objecter ms_handle_connect 0x7f98b406de60 2024-03-29T11:28:03.922 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.933+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98cc050250 con 0x7f98d00a9d70 2024-03-29T11:28:03.922 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.933+0000 7f98c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:03.922 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:03.933+0000 7f98c67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:04.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.322+0000 7f98d7ded640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:04.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.322+0000 7f98d7ded640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:04.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.322+0000 7f98d7ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:04.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.322+0000 7f98d7ded640 1 -- 172.21.15.59:0/118348871 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98b8005330 con 0x7f98d00a9d70 2024-03-29T11:28:04.312 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.323+0000 7f98c67fc640 1 -- 172.21.15.59:0/118348871 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f98cc052180 con 0x7f98d00a9d70 2024-03-29T11:28:04.312 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.323+0000 7f98c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:04.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.323+0000 7f98c67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:04.313 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:28:04.313 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:04.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 10 librados: watch_flush enter 2024-03-29T11:28:04.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 10 librados: watch_flush exit 2024-03-29T11:28:04.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-29T11:28:04.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 10 client.4522.objecter successfully canceled tick 2024-03-29T11:28:04.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 1 -- 172.21.15.59:0/118348871 >> v1:172.21.15.172:6816/2420742773 conn(0x7f98b406de60 legacy=0x7f98b4070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 10 monclient: shutdown 2024-03-29T11:28:04.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:04.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 1 -- 172.21.15.59:0/118348871 >> v1:172.21.15.172:6789/0 conn(0x7f98d00a9d70 legacy=0x7f98d006e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.317 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 1 -- 172.21.15.59:0/118348871 shutdown_connections 2024-03-29T11:28:04.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 1 -- 172.21.15.59:0/118348871 >> 172.21.15.59:0/118348871 conn(0x7f98d0066a00 msgr2=0x7f98d00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:04.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 1 -- 172.21.15.59:0/118348871 shutdown_connections 2024-03-29T11:28:04.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.328+0000 7f98d7ded640 1 -- 172.21.15.59:0/118348871 wait complete. 2024-03-29T11:28:04.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:04.329+0000 7f98d7ded640 1 librados: shutdown 2024-03-29T11:28:04.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:11.145 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:11.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.302+0000 7f15824f6640 10 monclient: get_monmap_and_config 2024-03-29T11:28:11.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.302+0000 7f15824f6640 10 monclient: build_initial_monmap 2024-03-29T11:28:11.293 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.302+0000 7f15824f6640 10 monclient: monmap: 2024-03-29T11:28:11.294 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:11.294 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:11.294 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:28:11.303418+0000 2024-03-29T11:28:11.294 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:28:11.303418+0000 2024-03-29T11:28:11.294 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:11.294 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:11.294 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:11.294 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:28:11.294 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:28:11.294 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:11.294 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.304+0000 7f15824f6640 1 Processor -- start 2024-03-29T11:28:11.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.305+0000 7f15824f6640 1 -- start start 2024-03-29T11:28:11.295 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.305+0000 7f15824f6640 10 monclient: init 2024-03-29T11:28:11.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.305+0000 7f15824f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:11.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.306+0000 7f15824f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:11.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.306+0000 7f15824f6640 10 monclient(hunting): picked mon.noname-c con 0x7f157c0a9ac0 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:11.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.306+0000 7f15824f6640 10 monclient(hunting): picked mon.noname-b con 0x7f157c13ee90 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:11.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.306+0000 7f15824f6640 10 monclient(hunting): picked mon.noname-a con 0x7f157c0af900 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:11.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.306+0000 7f15824f6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157c0afd10 con 0x7f157c0af900 2024-03-29T11:28:11.298 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.306+0000 7f15824f6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157c0af6f0 con 0x7f157c0a9ac0 2024-03-29T11:28:11.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.306+0000 7f15824f6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157c0aac80 con 0x7f157c13ee90 2024-03-29T11:28:11.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.306+0000 7f15824f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:11.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.306+0000 7f15824f6640 10 monclient(hunting): authenticate will time out at 1341.419769s 2024-03-29T11:28:11.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.309+0000 7f157b7fe640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f157c13ee90 0x7f157c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:54704/0 (socket says 172.21.15.59:54704) 2024-03-29T11:28:11.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.309+0000 7f157b7fe640 1 -- 172.21.15.59:0/3775269108 learned_addr learned my addr 172.21.15.59:0/3775269108 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:11.299 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 1 -- 172.21.15.59:0/3775269108 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964390416 0 0) 0x7f157c0aac80 con 0x7f157c13ee90 2024-03-29T11:28:11.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:11.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:11.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:11.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:28:11.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 1 -- 172.21.15.59:0/3775269108 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1564003520 con 0x7f157c13ee90 2024-03-29T11:28:11.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 1 -- 172.21.15.59:0/3775269108 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303915146 0 0) 0x7f157c0af6f0 con 0x7f157c0a9ac0 2024-03-29T11:28:11.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:11.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:11.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:11.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:28:11.300 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.310+0000 7f157affd640 1 -- 172.21.15.59:0/3775269108 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f157c0aac80 con 0x7f157c0a9ac0 2024-03-29T11:28:11.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 1 -- 172.21.15.59:0/3775269108 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3484276773 0 0) 0x7f1564003520 con 0x7f157c13ee90 2024-03-29T11:28:11.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 1 -- 172.21.15.59:0/3775269108 >> v1:172.21.15.172:6789/0 conn(0x7f157c0a9ac0 legacy=0x7f157c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 1 -- 172.21.15.59:0/3775269108 >> v1:172.21.15.59:6789/0 conn(0x7f157c0af900 legacy=0x7f157c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:11.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:11.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 1 monclient: found mon.noname-b 2024-03-29T11:28:11.304 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 1 -- 172.21.15.59:0/3775269108 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c0aaf60 con 0x7f157c13ee90 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 10 monclient: _finish_auth 0 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 1 -- 172.21.15.59:0/3775269108 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1570001690 con 0x7f157c13ee90 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f157affd640 10 monclient: dump: 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:11.305 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f15824f6640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.312+0000 7f15824f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.313+0000 7f157affd640 1 -- 172.21.15.59:0/3775269108 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15700017f0 con 0x7f157c13ee90 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.313+0000 7f157affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.313+0000 7f15824f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:11.306 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.313+0000 7f15824f6640 4 set_mon_vals no callback set 2024-03-29T11:28:11.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.314+0000 7f15824f6640 10 monclient: shutdown 2024-03-29T11:28:11.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.314+0000 7f15824f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:11.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.314+0000 7f15824f6640 1 -- 172.21.15.59:0/3775269108 >> v1:172.21.15.59:6790/0 conn(0x7f157c13ee90 legacy=0x7f157c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.314+0000 7f15824f6640 1 -- 172.21.15.59:0/3775269108 shutdown_connections 2024-03-29T11:28:11.307 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.314+0000 7f15824f6640 1 -- 172.21.15.59:0/3775269108 >> 172.21.15.59:0/3775269108 conn(0x7f157c0b35e0 msgr2=0x7f157c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:11.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.314+0000 7f15824f6640 1 -- 172.21.15.59:0/3775269108 shutdown_connections 2024-03-29T11:28:11.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.315+0000 7f15824f6640 1 -- 172.21.15.59:0/3775269108 wait complete. 2024-03-29T11:28:11.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.316+0000 7f15824f6640 10 monclient: build_initial_monmap 2024-03-29T11:28:11.310 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.316+0000 7f15824f6640 10 monclient: monmap: 2024-03-29T11:28:11.310 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:11.310 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:11.310 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:28:11.310 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.316+0000 7f15824f6640 1 librados: starting msgr at 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.316+0000 7f15824f6640 1 librados: starting objecter 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.316+0000 7f15824f6640 1 Processor -- start 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 1 -- start start 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 1 librados: setting wanted keys 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 1 librados: calling monclient init 2024-03-29T11:28:11.311 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 10 monclient: init 2024-03-29T11:28:11.312 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:11.312 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:11.312 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 10 monclient(hunting): picked mon.noname-b con 0x7f157c0a9ac0 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:11.312 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 10 monclient(hunting): picked mon.noname-a con 0x7f157c0af900 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:11.312 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 10 monclient(hunting): picked mon.noname-c con 0x7f157c13ee90 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:11.312 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157c1dc560 con 0x7f157c0af900 2024-03-29T11:28:11.312 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157c1db930 con 0x7f157c0a9ac0 2024-03-29T11:28:11.312 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157c141900 con 0x7f157c13ee90 2024-03-29T11:28:11.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:11.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.317+0000 7f15824f6640 10 monclient(hunting): authenticate will time out at 1341.431628s 2024-03-29T11:28:11.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.319+0000 7f1580a93640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f157c13ee90 0x7f157c1e60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:57854/0 (socket says 172.21.15.59:57854) 2024-03-29T11:28:11.313 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.319+0000 7f1580a93640 1 -- 172.21.15.59:0/3786996088 learned_addr learned my addr 172.21.15.59:0/3786996088 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:11.318 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.322+0000 7f1578ff9640 10 client.?.objecter ms_handle_connect 0x7f157c13ee90 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 10 client.?.objecter ms_handle_connect 0x7f157c0af900 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894708039 0 0) 0x7f157c141900 con 0x7f157c13ee90 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1568004eb0 con 0x7f157c13ee90 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645503444 0 0) 0x7f157c1dc560 con 0x7f157c0af900 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:11.319 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.323+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f157c141900 con 0x7f157c0af900 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1830661012 0 0) 0x7f1568004eb0 con 0x7f157c13ee90 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f157c1dc560 con 0x7f157c13ee90 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1574001c60 con 0x7f157c13ee90 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:11.320 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:11.321 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:11.321 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:11.321 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:11.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:11.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2497799724 0 0) 0x7f157c141900 con 0x7f157c0af900 2024-03-29T11:28:11.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1568004eb0 con 0x7f157c0af900 2024-03-29T11:28:11.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1570001650 con 0x7f157c0af900 2024-03-29T11:28:11.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:11.321 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:11.322 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.324+0000 7f1578ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:11.322 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:11.322 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:11.322 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:11.322 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:11.322 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:11.322 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:11.322 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.325+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2227945106 0 0) 0x7f1568004eb0 con 0x7f157c0af900 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.325+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 >> v1:172.21.15.59:6790/0 conn(0x7f157c13ee90 legacy=0x7f157c1e60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.325+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 >> v1:172.21.15.172:6789/0 conn(0x7f157c0a9ac0 legacy=0x7f157c1420f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 1 monclient: found mon.a 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c06ea10 con 0x7f157c0af900 2024-03-29T11:28:11.323 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 10 client.?.objecter ms_handle_connect 0x7f157c0a9ac0 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f15824f6640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f15824f6640 10 monclient: _renew_subs 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f15824f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f15824f6640 1 -- 172.21.15.59:0/3786996088 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157c06ecc0 con 0x7f157c0af900 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f15824f6640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f15824f6640 10 monclient: _renew_subs 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f15824f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f15824f6640 1 -- 172.21.15.59:0/3786996088 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f157c13f7a0 con 0x7f157c0af900 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f15824f6640 1 librados: init done 2024-03-29T11:28:11.324 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1570004df0 con 0x7f157c0af900 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1570005100 con 0x7f157c0af900 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f1578ff9640 10 monclient: dump: 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:11.325 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:11.326 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.326+0000 7f157a7fc640 4 set_mon_vals no callback set 2024-03-29T11:28:11.326 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.328+0000 7f15824f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:11.326 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.328+0000 7f15824f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:11.326 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.328+0000 7f15824f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:28:11.326 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.328+0000 7f15824f6640 1 -- 172.21.15.59:0/3786996088 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155c005080 con 0x7f157c0af900 2024-03-29T11:28:11.327 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.331+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f1570001aa0 con 0x7f157c0af900 2024-03-29T11:28:11.327 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.332+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f1570084b00 con 0x7f157c0af900 2024-03-29T11:28:11.327 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.332+0000 7f1578ff9640 10 client.4536.objecter ms_dispatch 0x7f157c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:11.328 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.332+0000 7f1578ff9640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:11.328 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.333+0000 7f1578ff9640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:11.328 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.333+0000 7f1578ff9640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-29T11:28:11.328 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.333+0000 7f157bfff640 10 monclient: get_auth_request con 0x7f156806de20 auth_method 0 2024-03-29T11:28:11.328 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.334+0000 7f1578ff9640 10 client.4536.objecter ms_handle_connect 0x7f156806de20 2024-03-29T11:28:11.328 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.338+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15700503e0 con 0x7f157c0af900 2024-03-29T11:28:11.328 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.338+0000 7f1578ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:11.328 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.338+0000 7f1578ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:11.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.781+0000 7f15824f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.781+0000 7f15824f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.781+0000 7f15824f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:28:11.771 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.781+0000 7f15824f6640 1 -- 172.21.15.59:0/3786996088 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f155c005330 con 0x7f157c0af900 2024-03-29T11:28:11.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.784+0000 7f1578ff9640 1 -- 172.21.15.59:0/3786996088 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f1570052310 con 0x7f157c0af900 2024-03-29T11:28:11.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.784+0000 7f1578ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.784+0000 7f1578ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:11.773 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:28:11.773 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:11.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.787+0000 7f15824f6640 10 librados: watch_flush enter 2024-03-29T11:28:11.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.787+0000 7f15824f6640 10 librados: watch_flush exit 2024-03-29T11:28:11.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.787+0000 7f15824f6640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-29T11:28:11.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.787+0000 7f15824f6640 10 client.4536.objecter successfully canceled tick 2024-03-29T11:28:11.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.788+0000 7f15824f6640 1 -- 172.21.15.59:0/3786996088 >> v1:172.21.15.172:6816/2420742773 conn(0x7f156806de20 legacy=0x7f15680702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.788+0000 7f15824f6640 10 monclient: shutdown 2024-03-29T11:28:11.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.788+0000 7f15824f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:11.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.788+0000 7f15824f6640 1 -- 172.21.15.59:0/3786996088 >> v1:172.21.15.59:6789/0 conn(0x7f157c0af900 legacy=0x7f157c1dbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.788+0000 7f15824f6640 1 -- 172.21.15.59:0/3786996088 shutdown_connections 2024-03-29T11:28:11.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.788+0000 7f15824f6640 1 -- 172.21.15.59:0/3786996088 >> 172.21.15.59:0/3786996088 conn(0x7f157c0b35e0 msgr2=0x7f157c14b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:11.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.788+0000 7f15824f6640 1 -- 172.21.15.59:0/3786996088 shutdown_connections 2024-03-29T11:28:11.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.788+0000 7f15824f6640 1 -- 172.21.15.59:0/3786996088 wait complete. 2024-03-29T11:28:11.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:11.788+0000 7f15824f6640 1 librados: shutdown 2024-03-29T11:28:11.793 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:18.600 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:18.755 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.765+0000 7f5fdf8a8640 10 monclient: get_monmap_and_config 2024-03-29T11:28:18.755 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.765+0000 7f5fdf8a8640 10 monclient: build_initial_monmap 2024-03-29T11:28:18.756 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.765+0000 7f5fdf8a8640 10 monclient: monmap: 2024-03-29T11:28:18.756 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:18.756 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:18.756 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:28:18.765955+0000 2024-03-29T11:28:18.756 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:28:18.765955+0000 2024-03-29T11:28:18.756 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:18.756 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:18.756 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:18.756 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:28:18.756 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:28:18.756 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:18.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.767+0000 7f5fdf8a8640 1 Processor -- start 2024-03-29T11:28:18.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.768+0000 7f5fdf8a8640 1 -- start start 2024-03-29T11:28:18.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.768+0000 7f5fdf8a8640 10 monclient: init 2024-03-29T11:28:18.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.769+0000 7f5fdf8a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:18.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.769+0000 7f5fdf8a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:18.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.769+0000 7f5fdf8a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5fd80a9930 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:18.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.769+0000 7f5fdf8a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5fd8149dc0 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:18.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.769+0000 7f5fdf8a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5fd814ff40 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:18.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.769+0000 7f5fdf8a8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd80aae60 con 0x7f5fd814ff40 2024-03-29T11:28:18.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.769+0000 7f5fdf8a8640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd80b4310 con 0x7f5fd80a9930 2024-03-29T11:28:18.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.769+0000 7f5fdf8a8640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd80afdd0 con 0x7f5fd8149dc0 2024-03-29T11:28:18.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.769+0000 7f5fdf8a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:18.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.769+0000 7f5fdf8a8640 10 monclient(hunting): authenticate will time out at 1348.883196s 2024-03-29T11:28:18.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.775+0000 7f5fdde45640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f5fd814ff40 0x7f5fd8152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:52809/0 (socket says 172.21.15.59:52809) 2024-03-29T11:28:18.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fdde45640 1 -- 172.21.15.59:0/3709781051 learned_addr learned my addr 172.21.15.59:0/3709781051 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:18.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fdce43640 1 --1- 172.21.15.59:0/3709781051 >> v1:172.21.15.59:6790/0 conn(0x7f5fd8149dc0 0x7f5fd814a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:60337/0 (socket says 172.21.15.59:60337) 2024-03-29T11:28:18.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164534651 0 0) 0x7f5fd80b4310 con 0x7f5fd80a9930 2024-03-29T11:28:18.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fcffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.772 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fcffff640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc4003520 con 0x7f5fd80a9930 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308534092 0 0) 0x7f5fd80afdd0 con 0x7f5fd8149dc0 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fcffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.776+0000 7f5fcffff640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd80b4310 con 0x7f5fd8149dc0 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163422371 0 0) 0x7f5fd80aae60 con 0x7f5fd814ff40 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd80afdd0 con 0x7f5fd814ff40 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4018581605 0 0) 0x7f5fc4003520 con 0x7f5fd80a9930 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 >> v1:172.21.15.59:6790/0 conn(0x7f5fd8149dc0 legacy=0x7f5fd814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 >> v1:172.21.15.59:6789/0 conn(0x7f5fd814ff40 legacy=0x7f5fd8152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 1 monclient: found mon.noname-c 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd814a950 con 0x7f5fd80a9930 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient: _finish_auth 0 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:18.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5fd4001690 con 0x7f5fd80a9930 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient: dump: 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:18.775 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fdf8a8640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fdf8a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fd40017f0 con 0x7f5fd80a9930 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 1 -- 172.21.15.59:0/3709781051 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5fd4001c40 con 0x7f5fd80a9930 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fcffff640 10 monclient: dump: 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:18.776 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fdf8a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.777+0000 7f5fdf8a8640 4 set_mon_vals no callback set 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.778+0000 7f5fdf8a8640 10 monclient: shutdown 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.778+0000 7f5fdf8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.778+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/3709781051 >> v1:172.21.15.172:6789/0 conn(0x7f5fd80a9930 legacy=0x7f5fd8143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.779+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/3709781051 shutdown_connections 2024-03-29T11:28:18.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.779+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/3709781051 >> 172.21.15.59:0/3709781051 conn(0x7f5fd80b35e0 msgr2=0x7f5fd80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:18.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.780+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/3709781051 shutdown_connections 2024-03-29T11:28:18.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.780+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/3709781051 wait complete. 2024-03-29T11:28:18.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.781+0000 7f5fdf8a8640 10 monclient: build_initial_monmap 2024-03-29T11:28:18.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.781+0000 7f5fdf8a8640 10 monclient: monmap: 2024-03-29T11:28:18.778 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:18.778 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:18.778 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.781+0000 7f5fdf8a8640 1 librados: starting msgr at 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.781+0000 7f5fdf8a8640 1 librados: starting objecter 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 1 Processor -- start 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 1 -- start start 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 1 librados: setting wanted keys 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 1 librados: calling monclient init 2024-03-29T11:28:18.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 10 monclient: init 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5fd80a9930 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5fd8149dc0 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5fd814ff40 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd81447b0 con 0x7f5fd814ff40 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd8144a90 con 0x7f5fd80a9930 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd8144d70 con 0x7f5fd8149dc0 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:18.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.782+0000 7f5fdf8a8640 10 monclient(hunting): authenticate will time out at 1348.896628s 2024-03-29T11:28:18.781 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fdce43640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f5fd8149dc0 0x7f5fd8148e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:60437/0 (socket says 172.21.15.59:60437) 2024-03-29T11:28:18.781 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fdce43640 1 -- 172.21.15.59:0/2248561466 learned_addr learned my addr 172.21.15.59:0/2248561466 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:18.783 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fcdffb640 10 client.?.objecter ms_handle_connect 0x7f5fd80a9930 2024-03-29T11:28:18.783 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fcdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:18.783 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fcdffb640 10 client.?.objecter ms_handle_connect 0x7f5fd8149dc0 2024-03-29T11:28:18.783 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fcdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:18.783 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262030046 0 0) 0x7f5fd8144a90 con 0x7f5fd80a9930 2024-03-29T11:28:18.783 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fcdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.783 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.784 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.784 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fcdffb640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:28:18.784 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.792+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fbc004eb0 con 0x7f5fd80a9930 2024-03-29T11:28:18.785 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 client.?.objecter ms_handle_connect 0x7f5fd814ff40 2024-03-29T11:28:18.785 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:18.785 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1126752687 0 0) 0x7f5fbc004eb0 con 0x7f5fd80a9930 2024-03-29T11:28:18.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fd8144a90 con 0x7f5fd80a9930 2024-03-29T11:28:18.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895611102 0 0) 0x7f5fd8144d70 con 0x7f5fd8149dc0 2024-03-29T11:28:18.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fbc004eb0 con 0x7f5fd8149dc0 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357795136 0 0) 0x7f5fd81447b0 con 0x7f5fd814ff40 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd8144d70 con 0x7f5fd814ff40 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5fd4001b00 con 0x7f5fd80a9930 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): dump: 2024-03-29T11:28:18.787 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:18.790 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:18.790 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:18.790 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:18.790 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.790 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:18.790 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:18.790 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:18.790 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:18.790 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:18.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:18.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3570357168 0 0) 0x7f5fbc004eb0 con 0x7f5fd8149dc0 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fd81447b0 con 0x7f5fd8149dc0 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5fc80018b0 con 0x7f5fd8149dc0 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient(hunting): dump: 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:18.791 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:18.792 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:18.792 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:18.792 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2041032843 0 0) 0x7f5fd8144a90 con 0x7f5fd80a9930 2024-03-29T11:28:18.794 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 >> v1:172.21.15.59:6790/0 conn(0x7f5fd8149dc0 legacy=0x7f5fd8148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.794 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 >> v1:172.21.15.59:6789/0 conn(0x7f5fd814ff40 legacy=0x7f5fd8139100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.794 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:18.794 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 monclient: found mon.b 2024-03-29T11:28:18.794 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:18.794 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:18.794 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd813c380 con 0x7f5fd80a9930 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fdf8a8640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fdf8a8640 10 monclient: _renew_subs 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fdf8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.796+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/2248561466 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd8149690 con 0x7f5fd80a9930 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.797+0000 7f5fdf8a8640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.797+0000 7f5fdf8a8640 10 monclient: _renew_subs 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.797+0000 7f5fdf8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.797+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/2248561466 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd8145370 con 0x7f5fd80a9930 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.797+0000 7f5fdf8a8640 1 librados: init done 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.799+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fd4003940 con 0x7f5fd80a9930 2024-03-29T11:28:18.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.799+0000 7f5fcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.799+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5fd4003c50 con 0x7f5fd80a9930 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.799+0000 7f5fcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.799+0000 7f5fcdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.799+0000 7f5fcdffb640 10 monclient: dump: 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:18.796 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.799+0000 7f5fdf8a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.799+0000 7f5fdf8a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.799+0000 7f5fdf8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.799+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/2248561466 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd8145670 con 0x7f5fd80a9930 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.799+0000 7f5fceffd640 4 set_mon_vals no callback set 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.801+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f5fd4019c20 con 0x7f5fd80a9930 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.801+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f5fd4083960 con 0x7f5fd80a9930 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.801+0000 7f5fcdffb640 10 client.4534.objecter ms_dispatch 0x7f5fd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.801+0000 7f5fcdffb640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.801+0000 7f5fcdffb640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.801+0000 7f5fcdffb640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.803+0000 7f5fdce43640 10 monclient: get_auth_request con 0x7f5fbc06df70 auth_method 0 2024-03-29T11:28:18.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.803+0000 7f5fcdffb640 10 client.4534.objecter ms_handle_connect 0x7f5fbc06df70 2024-03-29T11:28:18.799 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.809+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5fd8145670 con 0x7f5fd80a9930 2024-03-29T11:28:18.799 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.810+0000 7f5fcdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.799 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:18.810+0000 7f5fcdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:19.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.141+0000 7f5fdf8a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:19.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.141+0000 7f5fdf8a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:19.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.141+0000 7f5fdf8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:19.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.141+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/2248561466 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fd8145670 con 0x7f5fd80a9930 2024-03-29T11:28:19.134 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.145+0000 7f5fcdffb640 1 -- 172.21.15.59:0/2248561466 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f5fd8145670 con 0x7f5fd80a9930 2024-03-29T11:28:19.135 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.145+0000 7f5fcdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:19.135 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.145+0000 7f5fcdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:19.136 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:28:19.137 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:19.138 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.149+0000 7f5fdf8a8640 10 librados: watch_flush enter 2024-03-29T11:28:19.139 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.149+0000 7f5fdf8a8640 10 librados: watch_flush exit 2024-03-29T11:28:19.139 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.149+0000 7f5fdf8a8640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-29T11:28:19.139 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.149+0000 7f5fdf8a8640 10 client.4534.objecter successfully canceled tick 2024-03-29T11:28:19.141 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.152+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/2248561466 >> v1:172.21.15.172:6816/2420742773 conn(0x7f5fbc06df70 legacy=0x7f5fbc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.142 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.152+0000 7f5fdf8a8640 10 monclient: shutdown 2024-03-29T11:28:19.142 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.152+0000 7f5fdf8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:19.142 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.152+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/2248561466 >> v1:172.21.15.172:6789/0 conn(0x7f5fd80a9930 legacy=0x7f5fd8148750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.142 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.152+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/2248561466 shutdown_connections 2024-03-29T11:28:19.142 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.152+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/2248561466 >> 172.21.15.59:0/2248561466 conn(0x7f5fd80b35e0 msgr2=0x7f5fd80b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:19.142 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.152+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/2248561466 shutdown_connections 2024-03-29T11:28:19.142 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.152+0000 7f5fdf8a8640 1 -- 172.21.15.59:0/2248561466 wait complete. 2024-03-29T11:28:19.142 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:19.153+0000 7f5fdf8a8640 1 librados: shutdown 2024-03-29T11:28:19.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:25.968 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:26.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.128+0000 7f38a3fff640 10 monclient: get_monmap_and_config 2024-03-29T11:28:26.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.128+0000 7f38a3fff640 10 monclient: build_initial_monmap 2024-03-29T11:28:26.119 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.128+0000 7f38a3fff640 10 monclient: monmap: 2024-03-29T11:28:26.119 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:26.119 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:26.119 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:28:26.128931+0000 2024-03-29T11:28:26.120 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:28:26.128931+0000 2024-03-29T11:28:26.120 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:26.120 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:26.120 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:26.120 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:28:26.120 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:28:26.120 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:26.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.131+0000 7f38a3fff640 1 Processor -- start 2024-03-29T11:28:26.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.131+0000 7f38a3fff640 1 -- start start 2024-03-29T11:28:26.121 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.131+0000 7f38a3fff640 10 monclient: init 2024-03-29T11:28:26.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.132+0000 7f38a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:26.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.132+0000 7f38a3fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:26.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.132+0000 7f38a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f389c006800 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:26.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.132+0000 7f38a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f389c006be0 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:26.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.132+0000 7f38a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f389c031b60 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:26.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.132+0000 7f38a3fff640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c029370 con 0x7f389c006be0 2024-03-29T11:28:26.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.132+0000 7f38a3fff640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c0295e0 con 0x7f389c031b60 2024-03-29T11:28:26.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.132+0000 7f38a3fff640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c031360 con 0x7f389c006800 2024-03-29T11:28:26.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.132+0000 7f38a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:26.122 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.132+0000 7f38a3fff640 10 monclient(hunting): authenticate will time out at 1356.245797s 2024-03-29T11:28:26.130 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.137+0000 7f38a37fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f389c031b60 0x7f389c0d30c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:36705/0 (socket says 172.21.15.59:36705) 2024-03-29T11:28:26.130 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.137+0000 7f38a37fe640 1 -- 172.21.15.59:0/4062778595 learned_addr learned my addr 172.21.15.59:0/4062778595 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:26.130 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 -- 172.21.15.59:0/4062778595 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854538340 0 0) 0x7f389c031360 con 0x7f389c006800 2024-03-29T11:28:26.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:26.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:26.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:26.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:28:26.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 -- 172.21.15.59:0/4062778595 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f388c003520 con 0x7f389c006800 2024-03-29T11:28:26.131 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 -- 172.21.15.59:0/4062778595 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954102673 0 0) 0x7f389c029370 con 0x7f389c006be0 2024-03-29T11:28:26.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:26.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:26.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:26.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:28:26.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 -- 172.21.15.59:0/4062778595 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389c031360 con 0x7f389c006be0 2024-03-29T11:28:26.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 -- 172.21.15.59:0/4062778595 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1246642302 0 0) 0x7f388c003520 con 0x7f389c006800 2024-03-29T11:28:26.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 -- 172.21.15.59:0/4062778595 >> v1:172.21.15.172:6789/0 conn(0x7f389c031b60 legacy=0x7f389c0d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.132 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 -- 172.21.15.59:0/4062778595 >> v1:172.21.15.59:6789/0 conn(0x7f389c006be0 legacy=0x7f389c0c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.133 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:26.133 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:28:26.134 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:26.134 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:28:26.134 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 -- 172.21.15.59:0/4062778595 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c031640 con 0x7f389c006800 2024-03-29T11:28:26.135 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:26.135 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:26.135 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:26.135 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 -- 172.21.15.59:0/4062778595 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f38a404f730 con 0x7f389c006800 2024-03-29T11:28:26.135 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.135 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:26.136 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient: dump: 2024-03-29T11:28:26.136 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:26.136 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:26.136 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:26.136 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:26.136 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.136 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:26.137 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:26.137 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:26.137 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:26.137 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:26.137 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a3fff640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:28:26.137 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:26.138 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 -- 172.21.15.59:0/4062778595 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38a40008d0 con 0x7f389c006800 2024-03-29T11:28:26.138 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:26.138 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 1 -- 172.21.15.59:0/4062778595 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f38a40758b0 con 0x7f389c006800 2024-03-29T11:28:26.138 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a3fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:26.138 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a3fff640 4 set_mon_vals no callback set 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.139+0000 7f38a1ffb640 10 monclient: dump: 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:26.139 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:26.140 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.140+0000 7f38a3fff640 10 monclient: shutdown 2024-03-29T11:28:26.140 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.140+0000 7f38a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:26.140 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.140+0000 7f38a3fff640 1 -- 172.21.15.59:0/4062778595 >> v1:172.21.15.59:6790/0 conn(0x7f389c006800 legacy=0x7f389c0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.140 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.141+0000 7f38a3fff640 1 -- 172.21.15.59:0/4062778595 shutdown_connections 2024-03-29T11:28:26.140 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.141+0000 7f38a3fff640 1 -- 172.21.15.59:0/4062778595 >> 172.21.15.59:0/4062778595 conn(0x7f389c0058f0 msgr2=0x7f389c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:26.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.142+0000 7f38a3fff640 1 -- 172.21.15.59:0/4062778595 shutdown_connections 2024-03-29T11:28:26.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.142+0000 7f38a3fff640 1 -- 172.21.15.59:0/4062778595 wait complete. 2024-03-29T11:28:26.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.143+0000 7f38a3fff640 10 monclient: build_initial_monmap 2024-03-29T11:28:26.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.143+0000 7f38a3fff640 10 monclient: monmap: 2024-03-29T11:28:26.147 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:26.147 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:26.147 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:28:26.147 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.144+0000 7f38a3fff640 1 librados: starting msgr at 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.144+0000 7f38a3fff640 1 librados: starting objecter 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.144+0000 7f38a3fff640 1 Processor -- start 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.144+0000 7f38a3fff640 1 -- start start 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.144+0000 7f38a3fff640 1 librados: setting wanted keys 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.144+0000 7f38a3fff640 1 librados: calling monclient init 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.144+0000 7f38a3fff640 10 monclient: init 2024-03-29T11:28:26.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.145+0000 7f38a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:26.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.145+0000 7f38a3fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:26.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.145+0000 7f38a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f389c006800 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:26.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.145+0000 7f38a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f389c031b60 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:26.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.145+0000 7f38a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f389c0ca090 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:26.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.145+0000 7f38a3fff640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c0c6ec0 con 0x7f389c031b60 2024-03-29T11:28:26.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.145+0000 7f38a3fff640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c0c7310 con 0x7f389c006800 2024-03-29T11:28:26.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.145+0000 7f38a3fff640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c0c8c10 con 0x7f389c0ca090 2024-03-29T11:28:26.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.145+0000 7f38a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:26.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.145+0000 7f38a3fff640 10 monclient(hunting): authenticate will time out at 1356.259162s 2024-03-29T11:28:26.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.148+0000 7f38a37fe640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f389c0ca090 0x7f389c0bf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:42825/0 (socket says 172.21.15.59:42825) 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.148+0000 7f38a37fe640 1 -- 172.21.15.59:0/557568626 learned_addr learned my addr 172.21.15.59:0/557568626 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.151+0000 7f388b7fe640 10 client.?.objecter ms_handle_connect 0x7f389c006800 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.151+0000 7f388b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.151+0000 7f388b7fe640 10 client.?.objecter ms_handle_connect 0x7f389c0ca090 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.151+0000 7f388b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610792441 0 0) 0x7f389c0c7310 con 0x7f389c006800 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3890004eb0 con 0x7f389c006800 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499768475 0 0) 0x7f389c0c8c10 con 0x7f389c0ca090 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:26.150 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389c0c7310 con 0x7f389c0ca090 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2385006163 0 0) 0x7f3890004eb0 con 0x7f389c006800 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3890005490 con 0x7f389c006800 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f38a404f400 con 0x7f389c006800 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): dump: 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:26.151 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2086956953 0 0) 0x7f3890005490 con 0x7f389c006800 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 >> v1:172.21.15.59:6790/0 conn(0x7f389c0ca090 legacy=0x7f389c0bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 >> v1:172.21.15.59:6789/0 conn(0x7f389c031b60 legacy=0x7f389c0c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 monclient: found mon.b 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:26.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c0c96d0 con 0x7f389c006800 2024-03-29T11:28:26.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:26.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:26.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.152+0000 7f388b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:26.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f38a3fff640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:28:26.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f38a3fff640 10 monclient: _renew_subs 2024-03-29T11:28:26.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f38a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:26.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f38a3fff640 1 -- 172.21.15.59:0/557568626 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389c0bfdd0 con 0x7f389c006800 2024-03-29T11:28:26.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f38a3fff640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:26.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f38a3fff640 10 monclient: _renew_subs 2024-03-29T11:28:26.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f38a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:26.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f38a3fff640 1 -- 172.21.15.59:0/557568626 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c0c77a0 con 0x7f389c006800 2024-03-29T11:28:26.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f38a3fff640 1 librados: init done 2024-03-29T11:28:26.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38a404fdd0 con 0x7f389c006800 2024-03-29T11:28:26.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f388b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:26.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f38a4067bf0 con 0x7f389c006800 2024-03-29T11:28:26.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f388b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f388b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:26.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.153+0000 7f388b7fe640 10 monclient: dump: 2024-03-29T11:28:26.154 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:26.155 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:26.155 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:26.155 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:26.155 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.155 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:26.155 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:26.155 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:26.155 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:26.155 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:26.155 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.154+0000 7f38a17fa640 4 set_mon_vals no callback set 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.156+0000 7f38a3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.156+0000 7f38a3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.156+0000 7f38a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.156+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f38a4067d50 con 0x7f389c006800 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.156+0000 7f38a3fff640 1 -- 172.21.15.59:0/557568626 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3884005080 con 0x7f389c006800 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.158+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f38a40fe500 con 0x7f389c006800 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.158+0000 7f388b7fe640 10 client.4540.objecter ms_dispatch 0x7f38a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.158+0000 7f388b7fe640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.158+0000 7f388b7fe640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.158+0000 7f388b7fe640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.159+0000 7f38a27fc640 10 monclient: get_auth_request con 0x7f389006dd90 auth_method 0 2024-03-29T11:28:26.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.161+0000 7f388b7fe640 10 client.4540.objecter ms_handle_connect 0x7f389006dd90 2024-03-29T11:28:26.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.163+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38a40c9de0 con 0x7f389c006800 2024-03-29T11:28:26.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.163+0000 7f388b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:26.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.163+0000 7f388b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:26.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.497+0000 7f38a3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:26.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.497+0000 7f38a3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:26.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.497+0000 7f38a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:28:26.487 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.497+0000 7f38a3fff640 1 -- 172.21.15.59:0/557568626 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3884005330 con 0x7f389c006800 2024-03-29T11:28:26.488 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.499+0000 7f388b7fe640 1 -- 172.21.15.59:0/557568626 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f38a40cbd10 con 0x7f389c006800 2024-03-29T11:28:26.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.499+0000 7f388b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:26.489 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.499+0000 7f388b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:26.491 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:28:26.492 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:26.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.506+0000 7f38a3fff640 10 librados: watch_flush enter 2024-03-29T11:28:26.495 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.506+0000 7f38a3fff640 10 librados: watch_flush exit 2024-03-29T11:28:26.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.506+0000 7f38a3fff640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-29T11:28:26.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.506+0000 7f38a3fff640 10 client.4540.objecter successfully canceled tick 2024-03-29T11:28:26.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.506+0000 7f38a3fff640 1 -- 172.21.15.59:0/557568626 >> v1:172.21.15.172:6816/2420742773 conn(0x7f389006dd90 legacy=0x7f3890070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.506+0000 7f38a3fff640 10 monclient: shutdown 2024-03-29T11:28:26.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.506+0000 7f38a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:26.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.506+0000 7f38a3fff640 1 -- 172.21.15.59:0/557568626 >> v1:172.21.15.172:6789/0 conn(0x7f389c006800 legacy=0x7f389c0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.498 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.508+0000 7f38a3fff640 1 -- 172.21.15.59:0/557568626 shutdown_connections 2024-03-29T11:28:26.498 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.509+0000 7f38a3fff640 1 -- 172.21.15.59:0/557568626 >> 172.21.15.59:0/557568626 conn(0x7f389c0058f0 msgr2=0x7f389c0382a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:26.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.509+0000 7f38a3fff640 1 -- 172.21.15.59:0/557568626 shutdown_connections 2024-03-29T11:28:26.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.509+0000 7f38a3fff640 1 -- 172.21.15.59:0/557568626 wait complete. 2024-03-29T11:28:26.500 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:26.511+0000 7f38a3fff640 1 librados: shutdown 2024-03-29T11:28:26.520 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:33.330 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:33.502 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.511+0000 7f70f2354640 10 monclient: get_monmap_and_config 2024-03-29T11:28:33.502 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.511+0000 7f70f2354640 10 monclient: build_initial_monmap 2024-03-29T11:28:33.502 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.511+0000 7f70f2354640 10 monclient: monmap: 2024-03-29T11:28:33.503 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:33.503 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:33.503 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:28:33.513676+0000 2024-03-29T11:28:33.503 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:28:33.513676+0000 2024-03-29T11:28:33.503 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:33.503 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:33.503 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:33.503 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:28:33.503 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:28:33.503 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:33.504 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.512+0000 7f70f2354640 1 Processor -- start 2024-03-29T11:28:33.504 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.513+0000 7f70f2354640 1 -- start start 2024-03-29T11:28:33.504 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.513+0000 7f70f2354640 10 monclient: init 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.513+0000 7f70f2354640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.513+0000 7f70f2354640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.513+0000 7f70f2354640 10 monclient(hunting): picked mon.noname-b con 0x7f70ec0b1f60 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.513+0000 7f70f2354640 10 monclient(hunting): picked mon.noname-a con 0x7f70ec071eb0 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.513+0000 7f70f2354640 10 monclient(hunting): picked mon.noname-c con 0x7f70ec143d10 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.514+0000 7f70f2354640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec06ef80 con 0x7f70ec071eb0 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.514+0000 7f70f2354640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec06f260 con 0x7f70ec143d10 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.514+0000 7f70f2354640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec0b40c0 con 0x7f70ec0b1f60 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.514+0000 7f70f2354640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.514+0000 7f70f2354640 10 monclient(hunting): authenticate will time out at 1363.628713s 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.514+0000 7f70ebfff640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f70ec0b1f60 0x7f70ec143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:43198/0 (socket says 172.21.15.59:43198) 2024-03-29T11:28:33.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.514+0000 7f70ebfff640 1 -- 172.21.15.59:0/3754190476 learned_addr learned my addr 172.21.15.59:0/3754190476 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:33.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 1 -- 172.21.15.59:0/3754190476 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47970547 0 0) 0x7f70ec0b40c0 con 0x7f70ec0b1f60 2024-03-29T11:28:33.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:28:33.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 1 -- 172.21.15.59:0/3754190476 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d4003520 con 0x7f70ec0b1f60 2024-03-29T11:28:33.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 1 -- 172.21.15.59:0/3754190476 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608635659 0 0) 0x7f70ec06f260 con 0x7f70ec143d10 2024-03-29T11:28:33.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:28:33.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.515+0000 7f70eaffd640 1 -- 172.21.15.59:0/3754190476 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70ec0b40c0 con 0x7f70ec143d10 2024-03-29T11:28:33.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.516+0000 7f70eaffd640 1 -- 172.21.15.59:0/3754190476 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3109705180 0 0) 0x7f70ec0b40c0 con 0x7f70ec143d10 2024-03-29T11:28:33.509 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.516+0000 7f70eaffd640 1 -- 172.21.15.59:0/3754190476 >> v1:172.21.15.59:6790/0 conn(0x7f70ec0b1f60 legacy=0x7f70ec143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.516+0000 7f70eaffd640 1 -- 172.21.15.59:0/3754190476 >> v1:172.21.15.59:6789/0 conn(0x7f70ec071eb0 legacy=0x7f70ec146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 1 monclient: found mon.noname-c 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 1 -- 172.21.15.59:0/3754190476 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec155d70 con 0x7f70ec143d10 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 1 -- 172.21.15.59:0/3754190476 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f70e40019f0 con 0x7f70ec143d10 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 10 monclient: dump: 2024-03-29T11:28:33.510 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70f2354640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70f2354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 1 -- 172.21.15.59:0/3754190476 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70e4001b50 con 0x7f70ec143d10 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:33.511 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 1 -- 172.21.15.59:0/3754190476 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f70e4001fa0 con 0x7f70ec143d10 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70eaffd640 10 monclient: dump: 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:33.512 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:33.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70f2354640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:33.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.517+0000 7f70f2354640 4 set_mon_vals no callback set 2024-03-29T11:28:33.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.518+0000 7f70f2354640 10 monclient: shutdown 2024-03-29T11:28:33.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.518+0000 7f70f2354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:33.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.518+0000 7f70f2354640 1 -- 172.21.15.59:0/3754190476 >> v1:172.21.15.172:6789/0 conn(0x7f70ec143d10 legacy=0x7f70ec146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.518+0000 7f70f2354640 1 -- 172.21.15.59:0/3754190476 shutdown_connections 2024-03-29T11:28:33.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.518+0000 7f70f2354640 1 -- 172.21.15.59:0/3754190476 >> 172.21.15.59:0/3754190476 conn(0x7f70ec066a00 msgr2=0x7f70ec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:33.518 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.521+0000 7f70f2354640 1 -- 172.21.15.59:0/3754190476 shutdown_connections 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.521+0000 7f70f2354640 1 -- 172.21.15.59:0/3754190476 wait complete. 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.522+0000 7f70f2354640 10 monclient: build_initial_monmap 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.522+0000 7f70f2354640 10 monclient: monmap: 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:28:33.519 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.522+0000 7f70f2354640 1 librados: starting msgr at 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.522+0000 7f70f2354640 1 librados: starting objecter 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.522+0000 7f70f2354640 1 Processor -- start 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.522+0000 7f70f2354640 1 -- start start 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.522+0000 7f70f2354640 1 librados: setting wanted keys 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.522+0000 7f70f2354640 1 librados: calling monclient init 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.522+0000 7f70f2354640 10 monclient: init 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f2354640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f2354640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f2354640 10 monclient(hunting): picked mon.noname-b con 0x7f70ec0b1f60 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f2354640 10 monclient(hunting): picked mon.noname-a con 0x7f70ec143d10 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f2354640 10 monclient(hunting): picked mon.noname-c con 0x7f70ec071550 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f2354640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec147ea0 con 0x7f70ec143d10 2024-03-29T11:28:33.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f2354640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec13b8d0 con 0x7f70ec0b1f60 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f2354640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec149a50 con 0x7f70ec071550 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f2354640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f2354640 10 monclient(hunting): authenticate will time out at 1363.638258s 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f08f1640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f70ec071550 0x7f70ec13b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:44696/0 (socket says 172.21.15.59:44696) 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70f08f1640 1 -- 172.21.15.59:0/3529286164 learned_addr learned my addr 172.21.15.59:0/3529286164 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70e8ff9640 10 client.?.objecter ms_handle_connect 0x7f70ec071550 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614237790 0 0) 0x7f70ec149a50 con 0x7f70ec071550 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70e8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70e8ff9640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.523+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d8004eb0 con 0x7f70ec071550 2024-03-29T11:28:33.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.524+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4139487572 0 0) 0x7f70d8004eb0 con 0x7f70ec071550 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.524+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70d8005300 con 0x7f70ec071550 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.524+0000 7f70e8ff9640 10 client.?.objecter ms_handle_connect 0x7f70ec0b1f60 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.524+0000 7f70e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.524+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f70e4001fa0 con 0x7f70ec071550 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.524+0000 7f70e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.524+0000 7f70e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.524+0000 7f70e8ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:33.522 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.524+0000 7f70e8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.525+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221464519 0 0) 0x7f70ec13b8d0 con 0x7f70ec0b1f60 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.525+0000 7f70e8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.525+0000 7f70e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.525+0000 7f70e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.525+0000 7f70e8ff9640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.525+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d8004eb0 con 0x7f70ec0b1f60 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.525+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1268629600 0 0) 0x7f70d8005300 con 0x7f70ec071550 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.525+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 >> v1:172.21.15.172:6789/0 conn(0x7f70ec0b1f60 legacy=0x7f70ec147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.525+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 >> v1:172.21.15.59:6789/0 conn(0x7f70ec143d10 legacy=0x7f70ec13aab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:33.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 1 monclient: found mon.c 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec1496e0 con 0x7f70ec071550 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70f2354640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70f2354640 10 monclient: _renew_subs 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70f2354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70f2354640 1 -- 172.21.15.59:0/3529286164 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ec149270 con 0x7f70ec071550 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70f2354640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70f2354640 10 monclient: _renew_subs 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70f2354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:33.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70f2354640 1 -- 172.21.15.59:0/3529286164 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ec070e60 con 0x7f70ec071550 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70f2354640 1 librados: init done 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70e4003c20 con 0x7f70ec071550 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f70e4002100 con 0x7f70ec071550 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70e8ff9640 10 monclient: dump: 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:33.525 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:33.526 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:33.526 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:33.526 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.527+0000 7f70ea7fc640 4 set_mon_vals no callback set 2024-03-29T11:28:33.527 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.529+0000 7f70f2354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:33.527 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.529+0000 7f70f2354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:33.527 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.529+0000 7f70f2354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:33.527 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.529+0000 7f70f2354640 1 -- 172.21.15.59:0/3529286164 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70cc005080 con 0x7f70ec071550 2024-03-29T11:28:33.527 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.530+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f70e4004070 con 0x7f70ec071550 2024-03-29T11:28:33.527 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.530+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f70e4084e10 con 0x7f70ec071550 2024-03-29T11:28:33.528 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.530+0000 7f70e8ff9640 10 client.4574.objecter ms_dispatch 0x7f70ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:33.528 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.530+0000 7f70e8ff9640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:33.528 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.530+0000 7f70e8ff9640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:33.528 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.530+0000 7f70e8ff9640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T11:28:33.528 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.530+0000 7f70ebfff640 10 monclient: get_auth_request con 0x7f70d806df40 auth_method 0 2024-03-29T11:28:33.528 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.530+0000 7f70e8ff9640 10 client.4574.objecter ms_handle_connect 0x7f70d806df40 2024-03-29T11:28:33.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.539+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70e4050750 con 0x7f70ec071550 2024-03-29T11:28:33.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.539+0000 7f70e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:33.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.539+0000 7f70e8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:33.842 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.851+0000 7f70f2354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.842 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.851+0000 7f70f2354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.842 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.851+0000 7f70f2354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:33.842 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.851+0000 7f70f2354640 1 -- 172.21.15.59:0/3529286164 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70cc005330 con 0x7f70ec071550 2024-03-29T11:28:33.843 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.852+0000 7f70e8ff9640 1 -- 172.21.15.59:0/3529286164 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f70e40844c0 con 0x7f70ec071550 2024-03-29T11:28:33.843 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.852+0000 7f70e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.843 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.852+0000 7f70e8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:33.844 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:28:33.844 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:33.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.855+0000 7f70f2354640 10 librados: watch_flush enter 2024-03-29T11:28:33.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.855+0000 7f70f2354640 10 librados: watch_flush exit 2024-03-29T11:28:33.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.855+0000 7f70f2354640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T11:28:33.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.855+0000 7f70f2354640 10 client.4574.objecter successfully canceled tick 2024-03-29T11:28:33.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.855+0000 7f70f2354640 1 -- 172.21.15.59:0/3529286164 >> v1:172.21.15.172:6816/2420742773 conn(0x7f70d806df40 legacy=0x7f70d8070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.855+0000 7f70f2354640 10 monclient: shutdown 2024-03-29T11:28:33.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.855+0000 7f70f2354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:33.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.855+0000 7f70f2354640 1 -- 172.21.15.59:0/3529286164 >> v1:172.21.15.59:6790/0 conn(0x7f70ec071550 legacy=0x7f70ec13b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.856+0000 7f70f2354640 1 -- 172.21.15.59:0/3529286164 shutdown_connections 2024-03-29T11:28:33.847 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.856+0000 7f70f2354640 1 -- 172.21.15.59:0/3529286164 >> 172.21.15.59:0/3529286164 conn(0x7f70ec066a00 msgr2=0x7f70ec0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:33.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.856+0000 7f70f2354640 1 -- 172.21.15.59:0/3529286164 shutdown_connections 2024-03-29T11:28:33.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.856+0000 7f70f2354640 1 -- 172.21.15.59:0/3529286164 wait complete. 2024-03-29T11:28:33.848 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:33.856+0000 7f70f2354640 1 librados: shutdown 2024-03-29T11:28:33.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:40.671 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:40.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.812+0000 7efd4396d640 10 monclient: get_monmap_and_config 2024-03-29T11:28:40.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.812+0000 7efd4396d640 10 monclient: build_initial_monmap 2024-03-29T11:28:40.803 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.812+0000 7efd4396d640 10 monclient: monmap: 2024-03-29T11:28:40.803 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:40.803 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:40.803 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:28:40.813944+0000 2024-03-29T11:28:40.803 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:28:40.813944+0000 2024-03-29T11:28:40.803 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:40.803 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:40.804 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:40.804 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:28:40.804 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:28:40.804 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:40.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.812+0000 7efd4396d640 1 Processor -- start 2024-03-29T11:28:40.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.813+0000 7efd4396d640 1 -- start start 2024-03-29T11:28:40.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.813+0000 7efd4396d640 10 monclient: init 2024-03-29T11:28:40.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd4396d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:40.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd4396d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd4396d640 10 monclient(hunting): picked mon.noname-a con 0x7efd3c1422b0 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd4396d640 10 monclient(hunting): picked mon.noname-c con 0x7efd3c142690 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd40f08640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7efd3c142690 0x7efd3c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:52948/0 (socket says 172.21.15.59:52948) 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd41f0a640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7efd3c0b17d0 0x7efd3c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:59398/0 (socket says 172.21.15.59:59398) 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd4396d640 10 monclient(hunting): picked mon.noname-b con 0x7efd3c0b17d0 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd40f08640 1 -- 172.21.15.59:0/1107148104 learned_addr learned my addr 172.21.15.59:0/1107148104 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd4396d640 1 -- 172.21.15.59:0/1107148104 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c142c80 con 0x7efd3c1422b0 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd4396d640 1 -- 172.21.15.59:0/1107148104 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0aae40 con 0x7efd3c142690 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd4396d640 1 -- 172.21.15.59:0/1107148104 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c13fbd0 con 0x7efd3c0b17d0 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd4396d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.814+0000 7efd4396d640 10 monclient(hunting): authenticate will time out at 1370.929643s 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 1 -- 172.21.15.59:0/1107148104 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409630004 0 0) 0x7efd3c0aae40 con 0x7efd3c142690 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:28:40.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 1 -- 172.21.15.59:0/1107148104 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd28003520 con 0x7efd3c142690 2024-03-29T11:28:40.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 1 -- 172.21.15.59:0/1107148104 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690387820 0 0) 0x7efd3c13fbd0 con 0x7efd3c0b17d0 2024-03-29T11:28:40.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:28:40.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 1 -- 172.21.15.59:0/1107148104 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c0aae40 con 0x7efd3c0b17d0 2024-03-29T11:28:40.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 1 -- 172.21.15.59:0/1107148104 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1081213460 0 0) 0x7efd28003520 con 0x7efd3c142690 2024-03-29T11:28:40.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 1 -- 172.21.15.59:0/1107148104 >> v1:172.21.15.59:6790/0 conn(0x7efd3c0b17d0 legacy=0x7efd3c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.815+0000 7efd33fff640 1 -- 172.21.15.59:0/1107148104 >> v1:172.21.15.59:6789/0 conn(0x7efd3c1422b0 legacy=0x7efd3c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:40.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:40.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 1 monclient: found mon.noname-c 2024-03-29T11:28:40.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:40.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:28:40.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 1 -- 172.21.15.59:0/1107148104 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c14f920 con 0x7efd3c142690 2024-03-29T11:28:40.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:40.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:40.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:40.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 1 -- 172.21.15.59:0/1107148104 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7efd2c001690 con 0x7efd3c142690 2024-03-29T11:28:40.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd33fff640 10 monclient: dump: 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd4396d640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.817+0000 7efd4396d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:40.812 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.818+0000 7efd33fff640 1 -- 172.21.15.59:0/1107148104 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd2c001870 con 0x7efd3c142690 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.818+0000 7efd33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.818+0000 7efd33fff640 1 -- 172.21.15.59:0/1107148104 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7efd2c001c00 con 0x7efd3c142690 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.818+0000 7efd33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.818+0000 7efd33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.818+0000 7efd33fff640 10 monclient: dump: 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:40.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.819+0000 7efd4396d640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:40.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.819+0000 7efd4396d640 4 set_mon_vals no callback set 2024-03-29T11:28:40.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.820+0000 7efd4396d640 10 monclient: shutdown 2024-03-29T11:28:40.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.820+0000 7efd4396d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:40.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.820+0000 7efd4396d640 1 -- 172.21.15.59:0/1107148104 >> v1:172.21.15.172:6789/0 conn(0x7efd3c142690 legacy=0x7efd3c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.820+0000 7efd4396d640 1 -- 172.21.15.59:0/1107148104 shutdown_connections 2024-03-29T11:28:40.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.820+0000 7efd4396d640 1 -- 172.21.15.59:0/1107148104 >> 172.21.15.59:0/1107148104 conn(0x7efd3c066a00 msgr2=0x7efd3c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:40.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.820+0000 7efd4396d640 1 -- 172.21.15.59:0/1107148104 shutdown_connections 2024-03-29T11:28:40.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.820+0000 7efd4396d640 1 -- 172.21.15.59:0/1107148104 wait complete. 2024-03-29T11:28:40.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.821+0000 7efd4396d640 10 monclient: build_initial_monmap 2024-03-29T11:28:40.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.822+0000 7efd4396d640 10 monclient: monmap: 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.822+0000 7efd4396d640 1 librados: starting msgr at 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.822+0000 7efd4396d640 1 librados: starting objecter 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.822+0000 7efd4396d640 1 Processor -- start 2024-03-29T11:28:40.818 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.823+0000 7efd4396d640 1 -- start start 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.823+0000 7efd4396d640 1 librados: setting wanted keys 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.823+0000 7efd4396d640 1 librados: calling monclient init 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.823+0000 7efd4396d640 10 monclient: init 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd4396d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd4396d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd4396d640 10 monclient(hunting): picked mon.noname-b con 0x7efd3c0b17d0 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd4396d640 10 monclient(hunting): picked mon.noname-c con 0x7efd3c1422b0 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd4396d640 10 monclient(hunting): picked mon.noname-a con 0x7efd3c142690 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd40f08640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7efd3c1422b0 0x7efd3c0708a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:44322/0 (socket says 172.21.15.59:44322) 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd40f08640 1 -- 172.21.15.59:0/3331397614 learned_addr learned my addr 172.21.15.59:0/3331397614 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c134150 con 0x7efd3c142690 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd31ffb640 10 client.?.objecter ms_handle_connect 0x7efd3c1422b0 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c133d90 con 0x7efd3c0b17d0 2024-03-29T11:28:40.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd31ffb640 10 client.?.objecter ms_handle_connect 0x7efd3c0b17d0 2024-03-29T11:28:40.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.824+0000 7efd31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:40.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.825+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c06de90 con 0x7efd3c1422b0 2024-03-29T11:28:40.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.825+0000 7efd4396d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:40.820 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.825+0000 7efd4396d640 10 monclient(hunting): authenticate will time out at 1370.940073s 2024-03-29T11:28:40.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.825+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268788832 0 0) 0x7efd3c06de90 con 0x7efd3c1422b0 2024-03-29T11:28:40.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.825+0000 7efd31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.825+0000 7efd31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.825+0000 7efd31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.825+0000 7efd31ffb640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:28:40.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.825+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd20004eb0 con 0x7efd3c1422b0 2024-03-29T11:28:40.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741935248 0 0) 0x7efd3c133d90 con 0x7efd3c0b17d0 2024-03-29T11:28:40.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.822 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c06de90 con 0x7efd3c0b17d0 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2207659422 0 0) 0x7efd20004eb0 con 0x7efd3c1422b0 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd3c133d90 con 0x7efd3c1422b0 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7efd2c001c40 con 0x7efd3c1422b0 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:40.823 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.826+0000 7efd31ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.827+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2850578877 0 0) 0x7efd3c133d90 con 0x7efd3c1422b0 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.827+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 >> v1:172.21.15.172:6789/0 conn(0x7efd3c0b17d0 legacy=0x7efd3c14d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.827+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 >> v1:172.21.15.59:6789/0 conn(0x7efd3c142690 legacy=0x7efd3c1e72f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 1 monclient: found mon.c 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c06e020 con 0x7efd3c1422b0 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:40.824 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd4396d640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd4396d640 10 monclient: _renew_subs 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd4396d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3c1defb0 con 0x7efd3c1422b0 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd2c004a90 con 0x7efd3c1422b0 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7efd2c004da0 con 0x7efd3c1422b0 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd31ffb640 10 monclient: dump: 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:40.825 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.828+0000 7efd337fe640 4 set_mon_vals no callback set 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.830+0000 7efd4396d640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.830+0000 7efd4396d640 10 monclient: _renew_subs 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.830+0000 7efd4396d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.830+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3c14de50 con 0x7efd3c1422b0 2024-03-29T11:28:40.826 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.830+0000 7efd4396d640 1 librados: init done 2024-03-29T11:28:40.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.831+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7efd2c01ad70 con 0x7efd3c1422b0 2024-03-29T11:28:40.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.831+0000 7efd4396d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.827 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.831+0000 7efd4396d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.831+0000 7efd4396d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:40.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.831+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3c133f20 con 0x7efd3c1422b0 2024-03-29T11:28:40.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.832+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7efd2c085c20 con 0x7efd3c1422b0 2024-03-29T11:28:40.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.832+0000 7efd31ffb640 10 client.4580.objecter ms_dispatch 0x7efd3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:40.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.832+0000 7efd31ffb640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:40.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.832+0000 7efd31ffb640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:40.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.832+0000 7efd31ffb640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T11:28:40.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.832+0000 7efd41709640 10 monclient: get_auth_request con 0x7efd2006dcb0 auth_method 0 2024-03-29T11:28:40.828 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.834+0000 7efd31ffb640 10 client.4580.objecter ms_handle_connect 0x7efd2006dcb0 2024-03-29T11:28:40.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.839+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd3c133f20 con 0x7efd3c1422b0 2024-03-29T11:28:40.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.839+0000 7efd31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.830 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:40.839+0000 7efd31ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:41.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.166+0000 7efd4396d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:41.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.166+0000 7efd4396d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:41.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.166+0000 7efd4396d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:41.157 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.166+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd3c133f20 con 0x7efd3c1422b0 2024-03-29T11:28:41.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.167+0000 7efd31ffb640 1 -- 172.21.15.59:0/3331397614 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7efd3c133f20 con 0x7efd3c1422b0 2024-03-29T11:28:41.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.167+0000 7efd31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:41.158 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.167+0000 7efd31ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:41.158 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:28:41.158 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:41.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.171+0000 7efd4396d640 10 librados: watch_flush enter 2024-03-29T11:28:41.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.171+0000 7efd4396d640 10 librados: watch_flush exit 2024-03-29T11:28:41.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.171+0000 7efd4396d640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T11:28:41.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.171+0000 7efd4396d640 10 client.4580.objecter successfully canceled tick 2024-03-29T11:28:41.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.171+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 >> v1:172.21.15.172:6816/2420742773 conn(0x7efd2006dcb0 legacy=0x7efd20070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:41.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.171+0000 7efd4396d640 10 monclient: shutdown 2024-03-29T11:28:41.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.171+0000 7efd4396d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:41.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.171+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 >> v1:172.21.15.59:6790/0 conn(0x7efd3c1422b0 legacy=0x7efd3c0708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:41.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.171+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 shutdown_connections 2024-03-29T11:28:41.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.171+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 >> 172.21.15.59:0/3331397614 conn(0x7efd3c066a00 msgr2=0x7efd3c13d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:41.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.172+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 shutdown_connections 2024-03-29T11:28:41.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.172+0000 7efd4396d640 1 -- 172.21.15.59:0/3331397614 wait complete. 2024-03-29T11:28:41.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:41.172+0000 7efd4396d640 1 librados: shutdown 2024-03-29T11:28:41.179 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:47.987 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:48.138 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.146+0000 7f4ab367b640 10 monclient: get_monmap_and_config 2024-03-29T11:28:48.138 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.146+0000 7f4ab367b640 10 monclient: build_initial_monmap 2024-03-29T11:28:48.138 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.146+0000 7f4ab367b640 10 monclient: monmap: 2024-03-29T11:28:48.138 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:48.139 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:48.139 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:28:48.148000+0000 2024-03-29T11:28:48.139 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:28:48.148000+0000 2024-03-29T11:28:48.139 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:48.139 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:48.139 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:48.139 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:28:48.139 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:28:48.139 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:48.144 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.147+0000 7f4ab367b640 1 Processor -- start 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.147+0000 7f4ab367b640 1 -- start start 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.147+0000 7f4ab367b640 10 monclient: init 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.148+0000 7f4ab367b640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.148+0000 7f4ab367b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.148+0000 7f4ab367b640 10 monclient(hunting): picked mon.noname-c con 0x7f4aac0a9930 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.148+0000 7f4ab367b640 10 monclient(hunting): picked mon.noname-a con 0x7f4aac149dc0 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.148+0000 7f4ab367b640 10 monclient(hunting): picked mon.noname-b con 0x7f4aac14ff40 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.148+0000 7f4ab367b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0aae60 con 0x7f4aac149dc0 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.148+0000 7f4ab367b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0b4310 con 0x7f4aac0a9930 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.148+0000 7f4ab367b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0afdd0 con 0x7f4aac14ff40 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.148+0000 7f4ab367b640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:48.145 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.148+0000 7f4ab367b640 10 monclient(hunting): authenticate will time out at 1378.263620s 2024-03-29T11:28:48.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.152+0000 7f4ab0c16640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4aac149dc0 0x7f4aac14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:45703/0 (socket says 172.21.15.59:45703) 2024-03-29T11:28:48.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.152+0000 7f4ab0c16640 1 -- 172.21.15.59:0/375007224 learned_addr learned my addr 172.21.15.59:0/375007224 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:48.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.153+0000 7f4aa3fff640 1 -- 172.21.15.59:0/375007224 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769943486 0 0) 0x7f4aac0aae60 con 0x7f4aac149dc0 2024-03-29T11:28:48.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.153+0000 7f4aa3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.153+0000 7f4aa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.153+0000 7f4aa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.153+0000 7f4aa3fff640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:28:48.146 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.153+0000 7f4aa3fff640 1 -- 172.21.15.59:0/375007224 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a98003520 con 0x7f4aac149dc0 2024-03-29T11:28:48.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 1 -- 172.21.15.59:0/375007224 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1904830169 0 0) 0x7f4a98003520 con 0x7f4aac149dc0 2024-03-29T11:28:48.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 1 -- 172.21.15.59:0/375007224 >> v1:172.21.15.59:6790/0 conn(0x7f4aac14ff40 legacy=0x7f4aac152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:48.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 1 -- 172.21.15.59:0/375007224 >> v1:172.21.15.172:6789/0 conn(0x7f4aac0a9930 legacy=0x7f4aac143a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:28:48.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:48.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 1 monclient: found mon.noname-a 2024-03-29T11:28:48.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:48.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:28:48.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 1 -- 172.21.15.59:0/375007224 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac14a950 con 0x7f4aac149dc0 2024-03-29T11:28:48.147 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4ab367b640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4ab367b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 1 -- 172.21.15.59:0/375007224 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4a9c001690 con 0x7f4aac149dc0 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4aa3fff640 10 monclient: dump: 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.148 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:48.149 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:48.149 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:48.149 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:48.149 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:48.149 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.154+0000 7f4ab367b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:48.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.158+0000 7f4aa3fff640 1 -- 172.21.15.59:0/375007224 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a9c001860 con 0x7f4aac149dc0 2024-03-29T11:28:48.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.158+0000 7f4aa3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:48.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.158+0000 7f4aa3fff640 1 -- 172.21.15.59:0/375007224 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4a9c001cb0 con 0x7f4aac149dc0 2024-03-29T11:28:48.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.158+0000 7f4aa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.158+0000 7f4aa3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:48.151 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.158+0000 7f4aa3fff640 10 monclient: dump: 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.159+0000 7f4ab367b640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.159+0000 7f4ab367b640 4 set_mon_vals no callback set 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.160+0000 7f4ab367b640 10 monclient: shutdown 2024-03-29T11:28:48.152 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.160+0000 7f4ab367b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:48.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.160+0000 7f4ab367b640 1 -- 172.21.15.59:0/375007224 >> v1:172.21.15.59:6789/0 conn(0x7f4aac149dc0 legacy=0x7f4aac14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.160+0000 7f4ab367b640 1 -- 172.21.15.59:0/375007224 shutdown_connections 2024-03-29T11:28:48.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.160+0000 7f4ab367b640 1 -- 172.21.15.59:0/375007224 >> 172.21.15.59:0/375007224 conn(0x7f4aac0b35e0 msgr2=0x7f4aac0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:48.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.160+0000 7f4ab367b640 1 -- 172.21.15.59:0/375007224 shutdown_connections 2024-03-29T11:28:48.153 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.160+0000 7f4ab367b640 1 -- 172.21.15.59:0/375007224 wait complete. 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.161+0000 7f4ab367b640 10 monclient: build_initial_monmap 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.161+0000 7f4ab367b640 10 monclient: monmap: 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:48.154 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.162+0000 7f4ab367b640 1 librados: starting msgr at 2024-03-29T11:28:48.155 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.162+0000 7f4ab367b640 1 librados: starting objecter 2024-03-29T11:28:48.155 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.162+0000 7f4ab367b640 1 Processor -- start 2024-03-29T11:28:48.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.162+0000 7f4ab367b640 1 -- start start 2024-03-29T11:28:48.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.162+0000 7f4ab367b640 1 librados: setting wanted keys 2024-03-29T11:28:48.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.162+0000 7f4ab367b640 1 librados: calling monclient init 2024-03-29T11:28:48.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.162+0000 7f4ab367b640 10 monclient: init 2024-03-29T11:28:48.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.163+0000 7f4ab367b640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:48.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.163+0000 7f4ab367b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:48.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.163+0000 7f4ab367b640 10 monclient(hunting): picked mon.noname-a con 0x7f4aac0a9930 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:48.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.163+0000 7f4ab367b640 10 monclient(hunting): picked mon.noname-c con 0x7f4aac149dc0 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:48.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.163+0000 7f4ab367b640 10 monclient(hunting): picked mon.noname-b con 0x7f4aac14ff40 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:48.160 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.163+0000 7f4ab367b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac13bd30 con 0x7f4aac0a9930 2024-03-29T11:28:48.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.163+0000 7f4ab367b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac145c00 con 0x7f4aac14ff40 2024-03-29T11:28:48.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.163+0000 7f4ab367b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac149100 con 0x7f4aac149dc0 2024-03-29T11:28:48.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.163+0000 7f4ab367b640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:48.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.163+0000 7f4ab367b640 10 monclient(hunting): authenticate will time out at 1378.278233s 2024-03-29T11:28:48.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.169+0000 7f4ab1417640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4aac0a9930 0x7f4aac1de9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:45897/0 (socket says 172.21.15.59:45897) 2024-03-29T11:28:48.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.169+0000 7f4ab1417640 1 -- 172.21.15.59:0/1728239777 learned_addr learned my addr 172.21.15.59:0/1728239777 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:48.161 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 client.?.objecter ms_handle_connect 0x7f4aac0a9930 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 client.?.objecter ms_handle_connect 0x7f4aac149dc0 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262986500 0 0) 0x7f4aac13bd30 con 0x7f4aac0a9930 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a90004eb0 con 0x7f4aac0a9930 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 client.?.objecter ms_handle_connect 0x7f4aac14ff40 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913293091 0 0) 0x7f4aac149100 con 0x7f4aac149dc0 2024-03-29T11:28:48.162 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:28:48.163 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.170+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aac13bd30 con 0x7f4aac149dc0 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030643080 0 0) 0x7f4aac145c00 con 0x7f4aac14ff40 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aac149100 con 0x7f4aac14ff40 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1869862397 0 0) 0x7f4a90004eb0 con 0x7f4aac0a9930 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4aac145c00 con 0x7f4aac0a9930 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4aa8001870 con 0x7f4aac0a9930 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:48.164 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2364369073 0 0) 0x7f4aac145c00 con 0x7f4aac0a9930 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 >> v1:172.21.15.59:6790/0 conn(0x7f4aac149dc0 legacy=0x7f4aac13af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 >> v1:172.21.15.172:6789/0 conn(0x7f4aac14ff40 legacy=0x7f4aac13b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.165 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 1 monclient: found mon.a 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac149290 con 0x7f4aac0a9930 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4aa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4ab367b640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4ab367b640 10 monclient: _renew_subs 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4ab367b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.171+0000 7f4ab367b640 1 -- 172.21.15.59:0/1728239777 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aac144b60 con 0x7f4aac0a9930 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.172+0000 7f4ab367b640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.172+0000 7f4ab367b640 10 monclient: _renew_subs 2024-03-29T11:28:48.166 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.172+0000 7f4ab367b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.172+0000 7f4ab367b640 1 -- 172.21.15.59:0/1728239777 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aac0ac4c0 con 0x7f4aac0a9930 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.172+0000 7f4ab367b640 1 librados: init done 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.172+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aa80019d0 con 0x7f4aac0a9930 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.172+0000 7f4aa1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.172+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f4aa8004aa0 con 0x7f4aac0a9930 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.172+0000 7f4aa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.172+0000 7f4aa1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.172+0000 7f4aa1ffb640 10 monclient: dump: 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:48.167 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:48.168 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:48.168 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:48.168 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:48.169 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.174+0000 7f4aa2ffd640 4 set_mon_vals no callback set 2024-03-29T11:28:48.169 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.175+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f4aa8004c00 con 0x7f4aac0a9930 2024-03-29T11:28:48.169 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.175+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f4aa8084aa0 con 0x7f4aac0a9930 2024-03-29T11:28:48.169 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.175+0000 7f4ab0c16640 10 monclient: get_auth_request con 0x7f4a9006df20 auth_method 0 2024-03-29T11:28:48.170 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.176+0000 7f4aa1ffb640 10 client.4551.objecter ms_dispatch 0x7f4aac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:48.170 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.176+0000 7f4aa1ffb640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:48.170 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.176+0000 7f4aa1ffb640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:48.170 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.176+0000 7f4aa1ffb640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-29T11:28:48.170 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.177+0000 7f4ab367b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:48.170 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.177+0000 7f4ab367b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:48.170 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.177+0000 7f4ab367b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:28:48.170 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.177+0000 7f4ab367b640 1 -- 172.21.15.59:0/1728239777 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8c005080 con 0x7f4aac0a9930 2024-03-29T11:28:48.172 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.180+0000 7f4aa1ffb640 10 client.4551.objecter ms_handle_connect 0x7f4a9006df20 2024-03-29T11:28:48.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.188+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4aa8050380 con 0x7f4aac0a9930 2024-03-29T11:28:48.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.188+0000 7f4aa1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:48.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.188+0000 7f4aa1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:48.504 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.513+0000 7f4ab367b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:48.504 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.513+0000 7f4ab367b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:48.504 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.513+0000 7f4ab367b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:28:48.504 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.513+0000 7f4ab367b640 1 -- 172.21.15.59:0/1728239777 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a8c005330 con 0x7f4aac0a9930 2024-03-29T11:28:48.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.514+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1728239777 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f4aa80522b0 con 0x7f4aac0a9930 2024-03-29T11:28:48.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.514+0000 7f4aa1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:48.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.514+0000 7f4aa1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:48.508 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:28:48.508 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 10 librados: watch_flush enter 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 10 librados: watch_flush exit 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 10 client.4551.objecter successfully canceled tick 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 1 -- 172.21.15.59:0/1728239777 >> v1:172.21.15.172:6816/2420742773 conn(0x7f4a9006df20 legacy=0x7f4a900703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 10 monclient: shutdown 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 1 -- 172.21.15.59:0/1728239777 >> v1:172.21.15.59:6789/0 conn(0x7f4aac0a9930 legacy=0x7f4aac1de9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 1 -- 172.21.15.59:0/1728239777 shutdown_connections 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 1 -- 172.21.15.59:0/1728239777 >> 172.21.15.59:0/1728239777 conn(0x7f4aac0b35e0 msgr2=0x7f4aac0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 1 -- 172.21.15.59:0/1728239777 shutdown_connections 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 1 -- 172.21.15.59:0/1728239777 wait complete. 2024-03-29T11:28:48.510 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:48.518+0000 7f4ab367b640 1 librados: shutdown 2024-03-29T11:28:48.526 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:55.336 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:55.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.501+0000 7f0e5df23640 10 monclient: get_monmap_and_config 2024-03-29T11:28:55.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.501+0000 7f0e5df23640 10 monclient: build_initial_monmap 2024-03-29T11:28:55.493 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.501+0000 7f0e5df23640 10 monclient: monmap: 2024-03-29T11:28:55.493 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:55.493 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:55.493 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:28:55.502303+0000 2024-03-29T11:28:55.494 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:28:55.502303+0000 2024-03-29T11:28:55.494 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:55.494 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:55.494 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:55.494 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:28:55.494 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:28:55.494 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:55.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.505+0000 7f0e5df23640 1 Processor -- start 2024-03-29T11:28:55.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.505+0000 7f0e5df23640 1 -- start start 2024-03-29T11:28:55.496 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.505+0000 7f0e5df23640 10 monclient: init 2024-03-29T11:28:55.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.506+0000 7f0e5df23640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:55.499 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.506+0000 7f0e5df23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:55.500 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.506+0000 7f0e5df23640 10 monclient(hunting): picked mon.noname-b con 0x7f0e580a9930 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:55.505 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e5df23640 10 monclient(hunting): picked mon.noname-c con 0x7f0e58149dc0 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e5df23640 10 monclient(hunting): picked mon.noname-a con 0x7f0e5814ff40 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e5df23640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e580aae60 con 0x7f0e5814ff40 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e5df23640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e580b4310 con 0x7f0e58149dc0 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e5df23640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e580afdd0 con 0x7f0e580a9930 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e5df23640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e5df23640 10 monclient(hunting): authenticate will time out at 1385.626963s 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e56ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0e58149dc0 0x7f0e5814a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:36467/0 (socket says 172.21.15.59:36467) 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e56ffd640 1 -- 172.21.15.59:0/2259740579 learned_addr learned my addr 172.21.15.59:0/2259740579 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 1 -- 172.21.15.59:0/2259740579 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158238502 0 0) 0x7f0e580b4310 con 0x7f0e58149dc0 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:55.506 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:28:55.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 1 -- 172.21.15.59:0/2259740579 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e40003520 con 0x7f0e58149dc0 2024-03-29T11:28:55.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 1 -- 172.21.15.59:0/2259740579 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122584968 0 0) 0x7f0e580afdd0 con 0x7f0e580a9930 2024-03-29T11:28:55.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:55.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:55.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:55.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:28:55.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 1 -- 172.21.15.59:0/2259740579 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e580b4310 con 0x7f0e580a9930 2024-03-29T11:28:55.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.513+0000 7f0e567fc640 1 -- 172.21.15.59:0/2259740579 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1397321178 0 0) 0x7f0e40003520 con 0x7f0e58149dc0 2024-03-29T11:28:55.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.514+0000 7f0e567fc640 1 -- 172.21.15.59:0/2259740579 >> v1:172.21.15.59:6790/0 conn(0x7f0e580a9930 legacy=0x7f0e58143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.507 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.514+0000 7f0e567fc640 1 -- 172.21.15.59:0/2259740579 >> v1:172.21.15.59:6789/0 conn(0x7f0e5814ff40 legacy=0x7f0e58152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:55.511 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:55.511 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 1 monclient: found mon.noname-c 2024-03-29T11:28:55.511 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 1 -- 172.21.15.59:0/2259740579 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5814a950 con 0x7f0e58149dc0 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 1 -- 172.21.15.59:0/2259740579 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0e4c001610 con 0x7f0e58149dc0 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 10 monclient: dump: 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.512 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e5df23640 5 monclient: authenticate success, global_id 4558 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e5df23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 1 -- 172.21.15.59:0/2259740579 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e4c0017f0 con 0x7f0e58149dc0 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.520+0000 7f0e567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.521+0000 7f0e567fc640 1 -- 172.21.15.59:0/2259740579 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0e4c001c40 con 0x7f0e58149dc0 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.521+0000 7f0e567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.521+0000 7f0e567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.521+0000 7f0e567fc640 10 monclient: dump: 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:55.513 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:55.514 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:55.514 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:55.514 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.514 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:55.514 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:55.514 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:55.514 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:55.514 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:55.514 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.521+0000 7f0e5df23640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:55.514 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.521+0000 7f0e5df23640 4 set_mon_vals no callback set 2024-03-29T11:28:55.515 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.522+0000 7f0e5df23640 10 monclient: shutdown 2024-03-29T11:28:55.515 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.522+0000 7f0e5df23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:55.515 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.522+0000 7f0e5df23640 1 -- 172.21.15.59:0/2259740579 >> v1:172.21.15.172:6789/0 conn(0x7f0e58149dc0 legacy=0x7f0e5814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.515 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.525+0000 7f0e5df23640 1 -- 172.21.15.59:0/2259740579 shutdown_connections 2024-03-29T11:28:55.515 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.525+0000 7f0e5df23640 1 -- 172.21.15.59:0/2259740579 >> 172.21.15.59:0/2259740579 conn(0x7f0e580b35e0 msgr2=0x7f0e580b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:55.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.528+0000 7f0e5df23640 1 -- 172.21.15.59:0/2259740579 shutdown_connections 2024-03-29T11:28:55.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.528+0000 7f0e5df23640 1 -- 172.21.15.59:0/2259740579 wait complete. 2024-03-29T11:28:55.520 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.528+0000 7f0e5df23640 10 monclient: build_initial_monmap 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.528+0000 7f0e5df23640 10 monclient: monmap: 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.529+0000 7f0e5df23640 1 librados: starting msgr at 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.529+0000 7f0e5df23640 1 librados: starting objecter 2024-03-29T11:28:55.521 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.529+0000 7f0e5df23640 1 Processor -- start 2024-03-29T11:28:55.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.530+0000 7f0e5df23640 1 -- start start 2024-03-29T11:28:55.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.530+0000 7f0e5df23640 1 librados: setting wanted keys 2024-03-29T11:28:55.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.530+0000 7f0e5df23640 1 librados: calling monclient init 2024-03-29T11:28:55.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.530+0000 7f0e5df23640 10 monclient: init 2024-03-29T11:28:55.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.531+0000 7f0e5df23640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:55.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.531+0000 7f0e5df23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:55.522 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.531+0000 7f0e5df23640 10 monclient(hunting): picked mon.noname-c con 0x7f0e580a9930 addr v1:172.21.15.59:6790/0 2024-03-29T11:28:55.523 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.531+0000 7f0e5df23640 10 monclient(hunting): picked mon.noname-b con 0x7f0e58149dc0 addr v1:172.21.15.172:6789/0 2024-03-29T11:28:55.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.531+0000 7f0e5df23640 10 monclient(hunting): picked mon.noname-a con 0x7f0e5814ff40 addr v1:172.21.15.59:6789/0 2024-03-29T11:28:55.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.531+0000 7f0e5df23640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e58146020 con 0x7f0e5814ff40 2024-03-29T11:28:55.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.531+0000 7f0e5df23640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e581e3c20 con 0x7f0e58149dc0 2024-03-29T11:28:55.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.531+0000 7f0e5df23640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e58147ae0 con 0x7f0e580a9930 2024-03-29T11:28:55.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.531+0000 7f0e5df23640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:55.524 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.531+0000 7f0e5df23640 10 monclient(hunting): authenticate will time out at 1385.645462s 2024-03-29T11:28:55.525 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.535+0000 7f0e577fe640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f0e580a9930 0x7f0e58145910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:39437/0 (socket says 172.21.15.59:39437) 2024-03-29T11:28:55.526 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.535+0000 7f0e577fe640 1 -- 172.21.15.59:0/2026588969 learned_addr learned my addr 172.21.15.59:0/2026588969 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:28:55.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.537+0000 7f0e3ffff640 10 client.?.objecter ms_handle_connect 0x7f0e580a9930 2024-03-29T11:28:55.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.537+0000 7f0e3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:55.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.537+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882604 0 0) 0x7f0e58147ae0 con 0x7f0e580a9930 2024-03-29T11:28:55.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.537+0000 7f0e3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:55.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.537+0000 7f0e3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:55.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.537+0000 7f0e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:55.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.537+0000 7f0e3ffff640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:28:55.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.537+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e44004eb0 con 0x7f0e580a9930 2024-03-29T11:28:55.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.537+0000 7f0e3ffff640 10 client.?.objecter ms_handle_connect 0x7f0e5814ff40 2024-03-29T11:28:55.530 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.537+0000 7f0e3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.538+0000 7f0e3ffff640 10 client.?.objecter ms_handle_connect 0x7f0e58149dc0 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.538+0000 7f0e3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184412675 0 0) 0x7f0e581e3c20 con 0x7f0e58149dc0 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e58147ae0 con 0x7f0e58149dc0 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3897570329 0 0) 0x7f0e44004eb0 con 0x7f0e580a9930 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e44005490 con 0x7f0e580a9930 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0e50001d90 con 0x7f0e580a9930 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:55.531 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient(hunting): dump: 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1116242453 0 0) 0x7f0e44005490 con 0x7f0e580a9930 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 >> v1:172.21.15.172:6789/0 conn(0x7f0e58149dc0 legacy=0x7f0e581e0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 >> v1:172.21.15.59:6789/0 conn(0x7f0e5814ff40 legacy=0x7f0e581e40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.532 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 1 monclient: found mon.c 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e58144510 con 0x7f0e580a9930 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient: _finish_auth 0 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e5df23640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e5df23640 10 monclient: _renew_subs 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e5df23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e5df23640 1 -- 172.21.15.59:0/2026588969 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e581e15e0 con 0x7f0e580a9930 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e5df23640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e5df23640 10 monclient: _renew_subs 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e5df23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:55.533 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e5df23640 1 -- 172.21.15.59:0/2026588969 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e581e49a0 con 0x7f0e580a9930 2024-03-29T11:28:55.534 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.539+0000 7f0e5df23640 1 librados: init done 2024-03-29T11:28:55.535 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.541+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e50001ff0 con 0x7f0e580a9930 2024-03-29T11:28:55.535 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.541+0000 7f0e3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:55.535 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.541+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0e50003de0 con 0x7f0e580a9930 2024-03-29T11:28:55.535 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.541+0000 7f0e3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:55.535 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.541+0000 7f0e3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.541+0000 7f0e3ffff640 10 monclient: dump: 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.541+0000 7f0e55ffb640 4 set_mon_vals no callback set 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.542+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f0e50004000 con 0x7f0e580a9930 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.543+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f0e50084ea0 con 0x7f0e580a9930 2024-03-29T11:28:55.536 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.543+0000 7f0e3ffff640 10 client.4589.objecter ms_dispatch 0x7f0e58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:55.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.543+0000 7f0e3ffff640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:55.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.543+0000 7f0e3ffff640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:55.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.543+0000 7f0e3ffff640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:28:55.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.544+0000 7f0e5df23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:55.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.544+0000 7f0e5df23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:55.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.544+0000 7f0e5df23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:55.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.544+0000 7f0e5df23640 1 -- 172.21.15.59:0/2026588969 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e38005080 con 0x7f0e580a9930 2024-03-29T11:28:55.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.545+0000 7f0e56ffd640 10 monclient: get_auth_request con 0x7f0e4406df50 auth_method 0 2024-03-29T11:28:55.537 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.545+0000 7f0e3ffff640 10 client.4589.objecter ms_handle_connect 0x7f0e4406df50 2024-03-29T11:28:55.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.553+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e50050780 con 0x7f0e580a9930 2024-03-29T11:28:55.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.553+0000 7f0e3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:55.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.553+0000 7f0e3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:55.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.874+0000 7f0e5df23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:55.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.874+0000 7f0e5df23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:55.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.874+0000 7f0e5df23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:28:55.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.874+0000 7f0e5df23640 1 -- 172.21.15.59:0/2026588969 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e38005330 con 0x7f0e580a9930 2024-03-29T11:28:55.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.877+0000 7f0e3ffff640 1 -- 172.21.15.59:0/2026588969 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f0e500526b0 con 0x7f0e580a9930 2024-03-29T11:28:55.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.877+0000 7f0e3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:55.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.877+0000 7f0e3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:55.869 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:28:55.869 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:55.874 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.884+0000 7f0e5df23640 10 librados: watch_flush enter 2024-03-29T11:28:55.875 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.884+0000 7f0e5df23640 10 librados: watch_flush exit 2024-03-29T11:28:55.875 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.884+0000 7f0e5df23640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:28:55.875 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.884+0000 7f0e5df23640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:28:55.875 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.885+0000 7f0e5df23640 1 -- 172.21.15.59:0/2026588969 >> v1:172.21.15.172:6816/2420742773 conn(0x7f0e4406df50 legacy=0x7f0e44070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.885+0000 7f0e5df23640 10 monclient: shutdown 2024-03-29T11:28:55.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.885+0000 7f0e5df23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:55.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.885+0000 7f0e5df23640 1 -- 172.21.15.59:0/2026588969 >> v1:172.21.15.59:6790/0 conn(0x7f0e580a9930 legacy=0x7f0e58145910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.885+0000 7f0e5df23640 1 -- 172.21.15.59:0/2026588969 shutdown_connections 2024-03-29T11:28:55.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.885+0000 7f0e5df23640 1 -- 172.21.15.59:0/2026588969 >> 172.21.15.59:0/2026588969 conn(0x7f0e580b35e0 msgr2=0x7f0e580b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:55.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.885+0000 7f0e5df23640 1 -- 172.21.15.59:0/2026588969 shutdown_connections 2024-03-29T11:28:55.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.885+0000 7f0e5df23640 1 -- 172.21.15.59:0/2026588969 wait complete. 2024-03-29T11:28:55.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:28:55.885+0000 7f0e5df23640 1 librados: shutdown 2024-03-29T11:28:55.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:02.701 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:02.849 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.859+0000 7fc91359e640 10 monclient: get_monmap_and_config 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.859+0000 7fc91359e640 10 monclient: build_initial_monmap 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.859+0000 7fc91359e640 10 monclient: monmap: 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:29:02.859798+0000 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:29:02.859798+0000 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:29:02.850 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:02.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.860+0000 7fc91359e640 1 Processor -- start 2024-03-29T11:29:02.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.860+0000 7fc91359e640 1 -- start start 2024-03-29T11:29:02.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.860+0000 7fc91359e640 10 monclient: init 2024-03-29T11:29:02.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.861+0000 7fc91359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:02.851 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.861+0000 7fc91359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:02.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.861+0000 7fc91359e640 10 monclient(hunting): picked mon.noname-c con 0x7fc9140b4920 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:02.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.862+0000 7fc911d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc9140b4200 0x7fc9140b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59236/0 (socket says 172.21.15.59:59236) 2024-03-29T11:29:02.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.862+0000 7fc911d9b640 1 -- 172.21.15.59:0/2097055837 learned_addr learned my addr 172.21.15.59:0/2097055837 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:02.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.862+0000 7fc91359e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9140b4200 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:02.852 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.862+0000 7fc91359e640 10 monclient(hunting): picked mon.noname-b con 0x7fc91413f9a0 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:02.853 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.862+0000 7fc91359e640 1 -- 172.21.15.59:0/2097055837 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140b0930 con 0x7fc9140b4200 2024-03-29T11:29:02.853 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.862+0000 7fc91359e640 1 -- 172.21.15.59:0/2097055837 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140af460 con 0x7fc9140b4920 2024-03-29T11:29:02.853 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.862+0000 7fc91359e640 1 -- 172.21.15.59:0/2097055837 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140aeda0 con 0x7fc91413f9a0 2024-03-29T11:29:02.853 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.862+0000 7fc91359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:02.853 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.862+0000 7fc91359e640 10 monclient(hunting): authenticate will time out at 1392.976018s 2024-03-29T11:29:02.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.864+0000 7fc91159a640 1 -- 172.21.15.59:0/2097055837 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86669613 0 0) 0x7fc9140b0930 con 0x7fc9140b4200 2024-03-29T11:29:02.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.864+0000 7fc91159a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:02.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.864+0000 7fc91159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:02.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.864+0000 7fc91159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:02.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.864+0000 7fc91159a640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:29:02.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.864+0000 7fc91159a640 1 -- 172.21.15.59:0/2097055837 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc900003520 con 0x7fc9140b4200 2024-03-29T11:29:02.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.864+0000 7fc91159a640 1 -- 172.21.15.59:0/2097055837 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3821328180 0 0) 0x7fc900003520 con 0x7fc9140b4200 2024-03-29T11:29:02.855 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.864+0000 7fc91159a640 1 -- 172.21.15.59:0/2097055837 >> v1:172.21.15.59:6790/0 conn(0x7fc91413f9a0 legacy=0x7fc91414d8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:02.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 1 -- 172.21.15.59:0/2097055837 >> v1:172.21.15.172:6789/0 conn(0x7fc9140b4920 legacy=0x7fc91413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 10 monclient: _finish_hunting 0 2024-03-29T11:29:02.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 1 monclient: found mon.noname-a 2024-03-29T11:29:02.856 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 1 -- 172.21.15.59:0/2097055837 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9140af980 con 0x7fc9140b4200 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 10 monclient: _finish_auth 0 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 10 monclient: _check_auth_tickets 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91359e640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 1 -- 172.21.15.59:0/2097055837 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc904001690 con 0x7fc9140b4200 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91159a640 10 monclient: dump: 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:02.857 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:02.858 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:02.858 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:02.858 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:02.858 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:02.858 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:02.858 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:02.858 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:02.858 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:02.858 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.865+0000 7fc91359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:02.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.866+0000 7fc91159a640 1 -- 172.21.15.59:0/2097055837 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc904001920 con 0x7fc9140b4200 2024-03-29T11:29:02.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.866+0000 7fc91159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:02.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.866+0000 7fc91359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:02.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.866+0000 7fc91359e640 4 set_mon_vals no callback set 2024-03-29T11:29:02.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.867+0000 7fc91359e640 10 monclient: shutdown 2024-03-29T11:29:02.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.867+0000 7fc91359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:02.859 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.867+0000 7fc91359e640 1 -- 172.21.15.59:0/2097055837 >> v1:172.21.15.59:6789/0 conn(0x7fc9140b4200 legacy=0x7fc9140b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.861 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.871+0000 7fc91359e640 1 -- 172.21.15.59:0/2097055837 shutdown_connections 2024-03-29T11:29:02.861 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.871+0000 7fc91359e640 1 -- 172.21.15.59:0/2097055837 >> 172.21.15.59:0/2097055837 conn(0x7fc91406d230 msgr2=0x7fc914066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:02.861 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.871+0000 7fc91359e640 1 -- 172.21.15.59:0/2097055837 shutdown_connections 2024-03-29T11:29:02.862 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.871+0000 7fc91359e640 1 -- 172.21.15.59:0/2097055837 wait complete. 2024-03-29T11:29:02.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.873+0000 7fc91359e640 10 monclient: build_initial_monmap 2024-03-29T11:29:02.864 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.873+0000 7fc91359e640 10 monclient: monmap: 2024-03-29T11:29:02.864 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:02.864 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:02.864 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:29:02.864 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:29:02.864 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:02.864 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:02.864 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:02.864 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:29:02.864 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:29:02.865 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:02.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.873+0000 7fc91359e640 1 librados: starting msgr at 2024-03-29T11:29:02.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.873+0000 7fc91359e640 1 librados: starting objecter 2024-03-29T11:29:02.865 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.874+0000 7fc91359e640 1 Processor -- start 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.874+0000 7fc91359e640 1 -- start start 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.874+0000 7fc91359e640 1 librados: setting wanted keys 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.874+0000 7fc91359e640 1 librados: calling monclient init 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.874+0000 7fc91359e640 10 monclient: init 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.875+0000 7fc91359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.875+0000 7fc91359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.875+0000 7fc91359e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9140b4200 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.875+0000 7fc91359e640 10 monclient(hunting): picked mon.noname-c con 0x7fc9140b4920 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.875+0000 7fc91359e640 10 monclient(hunting): picked mon.noname-b con 0x7fc91413f9a0 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.876+0000 7fc91359e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc914142460 con 0x7fc9140b4200 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.876+0000 7fc91359e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140a9cf0 con 0x7fc91413f9a0 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.876+0000 7fc91359e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9141387a0 con 0x7fc9140b4920 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.876+0000 7fc91359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:02.867 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.876+0000 7fc91359e640 10 monclient(hunting): authenticate will time out at 1392.989950s 2024-03-29T11:29:02.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.876+0000 7fc91259c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc9140b4200 0x7fc9141d6ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40914/0 (socket says 172.21.15.59:40914) 2024-03-29T11:29:02.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.876+0000 7fc91259c640 1 -- 172.21.15.59:0/1763743052 learned_addr learned my addr 172.21.15.59:0/1763743052 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:02.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.878+0000 7fc90affd640 10 client.?.objecter ms_handle_connect 0x7fc9140b4200 2024-03-29T11:29:02.868 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.878+0000 7fc90affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:02.870 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.879+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540327510 0 0) 0x7fc914142460 con 0x7fc9140b4200 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.879+0000 7fc90affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.879+0000 7fc90affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.879+0000 7fc90affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.879+0000 7fc90affd640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.879+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f4004eb0 con 0x7fc9140b4200 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.879+0000 7fc90affd640 10 client.?.objecter ms_handle_connect 0x7fc91413f9a0 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.880+0000 7fc90affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.880+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824416825 0 0) 0x7fc9140a9cf0 con 0x7fc91413f9a0 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.880+0000 7fc90affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.880+0000 7fc90affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.880+0000 7fc90affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.880+0000 7fc90affd640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:29:02.871 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.880+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc914142460 con 0x7fc91413f9a0 2024-03-29T11:29:02.872 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.881+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1928453050 0 0) 0x7fc8f4004eb0 con 0x7fc9140b4200 2024-03-29T11:29:02.872 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.881+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9140a9cf0 con 0x7fc9140b4200 2024-03-29T11:29:02.872 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.881+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc8fc001930 con 0x7fc9140b4200 2024-03-29T11:29:02.872 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.881+0000 7fc90affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:02.872 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.881+0000 7fc90affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:02.872 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.881+0000 7fc90affd640 10 monclient(hunting): dump: 2024-03-29T11:29:02.872 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:02.872 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:02.873 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:02.873 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:02.873 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:02.873 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:02.873 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:02.873 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:02.873 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:02.873 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:02.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.881+0000 7fc90affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:02.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.881+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3699066938 0 0) 0x7fc9140a9cf0 con 0x7fc9140b4200 2024-03-29T11:29:02.873 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.881+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 >> v1:172.21.15.59:6790/0 conn(0x7fc9140b4920 legacy=0x7fc914137980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 >> v1:172.21.15.172:6789/0 conn(0x7fc91413f9a0 legacy=0x7fc914138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc90affd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc90affd640 1 monclient: found mon.a 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc90affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc90affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9141428f0 con 0x7fc9140b4200 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc90affd640 10 monclient: _finish_auth 0 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc90affd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc90affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc91359e640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc91359e640 10 monclient: _renew_subs 2024-03-29T11:29:02.876 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc91359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:02.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc91359e640 1 -- 172.21.15.59:0/1763743052 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc914141430 con 0x7fc9140b4200 2024-03-29T11:29:02.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc91359e640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:02.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc91359e640 10 monclient: _renew_subs 2024-03-29T11:29:02.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc91359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:02.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc91359e640 1 -- 172.21.15.59:0/1763743052 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9141d7490 con 0x7fc9140b4200 2024-03-29T11:29:02.877 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.885+0000 7fc91359e640 1 librados: init done 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.886+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8fc003610 con 0x7fc9140b4200 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.886+0000 7fc90affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.886+0000 7fc90bfff640 4 set_mon_vals no callback set 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.887+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fc8fc0039a0 con 0x7fc9140b4200 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.887+0000 7fc90affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.887+0000 7fc90affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.887+0000 7fc90affd640 10 monclient: dump: 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:02.878 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:02.879 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:02.879 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:02.879 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:02.879 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:02.879 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:02.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.887+0000 7fc91359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:02.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.887+0000 7fc91359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:02.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.888+0000 7fc91359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:02.879 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.888+0000 7fc91359e640 1 -- 172.21.15.59:0/1763743052 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f8005080 con 0x7fc9140b4200 2024-03-29T11:29:02.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.890+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fc8fc003b00 con 0x7fc9140b4200 2024-03-29T11:29:02.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.891+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fc8fc084a60 con 0x7fc9140b4200 2024-03-29T11:29:02.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.892+0000 7fc90affd640 10 client.4560.objecter ms_dispatch 0x7fc914000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:02.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.892+0000 7fc90affd640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:02.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.892+0000 7fc90affd640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:02.882 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.892+0000 7fc90affd640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T11:29:02.883 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.892+0000 7fc911d9b640 10 monclient: get_auth_request con 0x7fc8f406daf0 auth_method 0 2024-03-29T11:29:02.885 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.894+0000 7fc90affd640 10 client.4560.objecter ms_handle_connect 0x7fc8f406daf0 2024-03-29T11:29:02.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.896+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8fc050340 con 0x7fc9140b4200 2024-03-29T11:29:02.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.896+0000 7fc90affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:02.886 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:02.896+0000 7fc90affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:03.222 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.232+0000 7fc91359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:03.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.232+0000 7fc91359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:03.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.232+0000 7fc91359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:03.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.232+0000 7fc91359e640 1 -- 172.21.15.59:0/1763743052 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8f8005330 con 0x7fc9140b4200 2024-03-29T11:29:03.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.233+0000 7fc90affd640 1 -- 172.21.15.59:0/1763743052 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fc8fc052270 con 0x7fc9140b4200 2024-03-29T11:29:03.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.233+0000 7fc90affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:03.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.233+0000 7fc90affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:03.224 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:29:03.224 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:03.227 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 10 librados: watch_flush enter 2024-03-29T11:29:03.227 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 10 librados: watch_flush exit 2024-03-29T11:29:03.227 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T11:29:03.227 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 10 client.4560.objecter successfully canceled tick 2024-03-29T11:29:03.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 1 -- 172.21.15.59:0/1763743052 >> v1:172.21.15.172:6816/2420742773 conn(0x7fc8f406daf0 legacy=0x7fc8f406ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 10 monclient: shutdown 2024-03-29T11:29:03.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:03.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 1 -- 172.21.15.59:0/1763743052 >> v1:172.21.15.59:6789/0 conn(0x7fc9140b4200 legacy=0x7fc9141d6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 1 -- 172.21.15.59:0/1763743052 shutdown_connections 2024-03-29T11:29:03.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 1 -- 172.21.15.59:0/1763743052 >> 172.21.15.59:0/1763743052 conn(0x7fc91406d230 msgr2=0x7fc914069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:03.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 1 -- 172.21.15.59:0/1763743052 shutdown_connections 2024-03-29T11:29:03.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 1 -- 172.21.15.59:0/1763743052 wait complete. 2024-03-29T11:29:03.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:03.237+0000 7fc91359e640 1 librados: shutdown 2024-03-29T11:29:03.246 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:10.054 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:10.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.224+0000 7fcae049d640 10 monclient: get_monmap_and_config 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.224+0000 7fcae049d640 10 monclient: build_initial_monmap 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.224+0000 7fcae049d640 10 monclient: monmap: 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:29:10.225351+0000 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:29:10.225351+0000 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:29:10.216 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:10.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.226+0000 7fcae049d640 1 Processor -- start 2024-03-29T11:29:10.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.226+0000 7fcae049d640 1 -- start start 2024-03-29T11:29:10.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.226+0000 7fcae049d640 10 monclient: init 2024-03-29T11:29:10.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.227+0000 7fcae049d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:10.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.227+0000 7fcae049d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:10.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.227+0000 7fcae049d640 10 monclient(hunting): picked mon.noname-b con 0x7fcad80b0860 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:10.219 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.229+0000 7fcae049d640 10 monclient(hunting): picked mon.noname-a con 0x7fcad813ef30 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:10.222 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.229+0000 7fcae049d640 10 monclient(hunting): picked mon.noname-c con 0x7fcad80af8e0 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:10.222 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.229+0000 7fcae049d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad80b0c40 con 0x7fcad813ef30 2024-03-29T11:29:10.222 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.229+0000 7fcae049d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad80aed10 con 0x7fcad80af8e0 2024-03-29T11:29:10.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.229+0000 7fcae049d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad80afd50 con 0x7fcad80b0860 2024-03-29T11:29:10.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.229+0000 7fcae049d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:10.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.229+0000 7fcae049d640 10 monclient(hunting): authenticate will time out at 1400.343129s 2024-03-29T11:29:10.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.229+0000 7fcadda38640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcad813ef30 0x7fcad80af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:52776/0 (socket says 172.21.15.59:52776) 2024-03-29T11:29:10.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.229+0000 7fcadda38640 1 -- 172.21.15.59:0/1296578353 learned_addr learned my addr 172.21.15.59:0/1296578353 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:10.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.231+0000 7fcadd237640 1 -- 172.21.15.59:0/1296578353 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925196268 0 0) 0x7fcad80b0c40 con 0x7fcad813ef30 2024-03-29T11:29:10.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.231+0000 7fcadd237640 10 monclient(hunting): _negotiate 2024-03-29T11:29:10.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.231+0000 7fcadd237640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:10.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.231+0000 7fcadd237640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:10.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.231+0000 7fcadd237640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:29:10.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.231+0000 7fcadd237640 1 -- 172.21.15.59:0/1296578353 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac4007930 con 0x7fcad813ef30 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.232+0000 7fcadd237640 1 -- 172.21.15.59:0/1296578353 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4048027230 0 0) 0x7fcac4007930 con 0x7fcad813ef30 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.232+0000 7fcadd237640 1 -- 172.21.15.59:0/1296578353 >> v1:172.21.15.59:6790/0 conn(0x7fcad80b0860 legacy=0x7fcad813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 1 -- 172.21.15.59:0/1296578353 >> v1:172.21.15.172:6789/0 conn(0x7fcad80af8e0 legacy=0x7fcad8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 10 monclient: _finish_hunting 0 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 1 monclient: found mon.noname-a 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 1 -- 172.21.15.59:0/1296578353 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad80ab1b0 con 0x7fcad813ef30 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 10 monclient: _finish_auth 0 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 10 monclient: _check_auth_tickets 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 1 -- 172.21.15.59:0/1296578353 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fcad4001690 con 0x7fcad813ef30 2024-03-29T11:29:10.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 10 monclient: dump: 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:10.225 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:10.226 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.233+0000 7fcadd237640 1 -- 172.21.15.59:0/1296578353 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcad40018b0 con 0x7fcad813ef30 2024-03-29T11:29:10.226 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.234+0000 7fcadd237640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:10.226 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.234+0000 7fcadd237640 1 -- 172.21.15.59:0/1296578353 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fcad4001c40 con 0x7fcad813ef30 2024-03-29T11:29:10.226 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.234+0000 7fcadd237640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.226 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.234+0000 7fcadd237640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:10.226 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.234+0000 7fcadd237640 10 monclient: dump: 2024-03-29T11:29:10.226 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:10.226 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:10.226 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.234+0000 7fcae049d640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.234+0000 7fcae049d640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.234+0000 7fcae049d640 4 set_mon_vals no callback set 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.235+0000 7fcae049d640 10 monclient: shutdown 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.235+0000 7fcae049d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:10.227 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.235+0000 7fcae049d640 1 -- 172.21.15.59:0/1296578353 >> v1:172.21.15.59:6789/0 conn(0x7fcad813ef30 legacy=0x7fcad80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.237+0000 7fcae049d640 1 -- 172.21.15.59:0/1296578353 shutdown_connections 2024-03-29T11:29:10.228 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.237+0000 7fcae049d640 1 -- 172.21.15.59:0/1296578353 >> 172.21.15.59:0/1296578353 conn(0x7fcad806b660 msgr2=0x7fcad806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:10.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.238+0000 7fcae049d640 1 -- 172.21.15.59:0/1296578353 shutdown_connections 2024-03-29T11:29:10.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.238+0000 7fcae049d640 1 -- 172.21.15.59:0/1296578353 wait complete. 2024-03-29T11:29:10.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.239+0000 7fcae049d640 10 monclient: build_initial_monmap 2024-03-29T11:29:10.229 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.239+0000 7fcae049d640 10 monclient: monmap: 2024-03-29T11:29:10.230 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:10.230 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:10.230 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:29:10.230 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:29:10.230 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:10.230 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:10.230 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:10.230 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:29:10.230 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:29:10.230 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:10.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.240+0000 7fcae049d640 1 librados: starting msgr at 2024-03-29T11:29:10.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.240+0000 7fcae049d640 1 librados: starting objecter 2024-03-29T11:29:10.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.240+0000 7fcae049d640 1 Processor -- start 2024-03-29T11:29:10.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.240+0000 7fcae049d640 1 -- start start 2024-03-29T11:29:10.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.240+0000 7fcae049d640 1 librados: setting wanted keys 2024-03-29T11:29:10.231 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.240+0000 7fcae049d640 1 librados: calling monclient init 2024-03-29T11:29:10.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.240+0000 7fcae049d640 10 monclient: init 2024-03-29T11:29:10.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.241+0000 7fcae049d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:10.232 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.241+0000 7fcae049d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:10.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.244+0000 7fcae049d640 10 monclient(hunting): picked mon.noname-b con 0x7fcad80af8e0 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:10.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.245+0000 7fcae049d640 10 monclient(hunting): picked mon.noname-a con 0x7fcad80b0860 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:10.235 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.245+0000 7fcade239640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fcad80af8e0 0x7fcad81d3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:46315/0 (socket says 172.21.15.59:46315) 2024-03-29T11:29:10.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.245+0000 7fcadda38640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcad80b0860 0x7fcad8134350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:56900/0 (socket says 172.21.15.59:56900) 2024-03-29T11:29:10.237 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.246+0000 7fcae049d640 10 monclient(hunting): picked mon.noname-c con 0x7fcad813ef30 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:10.237 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.246+0000 7fcae049d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad80b3200 con 0x7fcad80b0860 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.246+0000 7fcae049d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad8135170 con 0x7fcad80af8e0 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.246+0000 7fcae049d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad80b2370 con 0x7fcad813ef30 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.246+0000 7fcae049d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.246+0000 7fcae049d640 10 monclient(hunting): authenticate will time out at 1400.359831s 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.246+0000 7fcade239640 1 -- 172.21.15.59:0/3036681906 learned_addr learned my addr 172.21.15.59:0/3036681906 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.247+0000 7fcaceffd640 10 client.?.objecter ms_handle_connect 0x7fcad80af8e0 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.247+0000 7fcaceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.247+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680717066 0 0) 0x7fcad8135170 con 0x7fcad80af8e0 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.247+0000 7fcaceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.247+0000 7fcaceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.247+0000 7fcaceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.247+0000 7fcaceffd640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:29:10.238 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.247+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcabc004eb0 con 0x7fcad80af8e0 2024-03-29T11:29:10.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.247+0000 7fcaceffd640 10 client.?.objecter ms_handle_connect 0x7fcad80b0860 2024-03-29T11:29:10.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.247+0000 7fcaceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:10.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 898695271 0 0) 0x7fcabc004eb0 con 0x7fcad80af8e0 2024-03-29T11:29:10.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcad8135170 con 0x7fcad80af8e0 2024-03-29T11:29:10.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fcac8001a30 con 0x7fcad80af8e0 2024-03-29T11:29:10.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:10.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 10 monclient(hunting): dump: 2024-03-29T11:29:10.241 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:10.241 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:10.241 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930627863 0 0) 0x7fcad80b3200 con 0x7fcad80b0860 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:29:10.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcabc004eb0 con 0x7fcad80b0860 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1296114537 0 0) 0x7fcad8135170 con 0x7fcad80af8e0 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 >> v1:172.21.15.59:6790/0 conn(0x7fcad813ef30 legacy=0x7fcad8134a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 >> v1:172.21.15.59:6789/0 conn(0x7fcad80b0860 legacy=0x7fcad8134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.248+0000 7fcaceffd640 1 monclient: found mon.b 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad80b3760 con 0x7fcad80af8e0 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcac8001cf0 con 0x7fcad80af8e0 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fcac8003a60 con 0x7fcad80af8e0 2024-03-29T11:29:10.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcaceffd640 10 monclient: dump: 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcadca36640 4 set_mon_vals no callback set 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcae049d640 5 monclient: authenticate success, global_id 4573 2024-03-29T11:29:10.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcae049d640 10 monclient: _renew_subs 2024-03-29T11:29:10.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcae049d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:29:10.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcae049d640 1 -- 172.21.15.59:0/3036681906 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad80b39b0 con 0x7fcad80af8e0 2024-03-29T11:29:10.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcae049d640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:10.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcae049d640 10 monclient: _renew_subs 2024-03-29T11:29:10.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcae049d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:29:10.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcae049d640 1 -- 172.21.15.59:0/3036681906 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad81c52a0 con 0x7fcad80af8e0 2024-03-29T11:29:10.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.249+0000 7fcae049d640 1 librados: init done 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.251+0000 7fcae049d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.251+0000 7fcae049d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.251+0000 7fcae049d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.251+0000 7fcae049d640 1 -- 172.21.15.59:0/3036681906 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad8075f10 con 0x7fcad80af8e0 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.252+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fcac8003bc0 con 0x7fcad80af8e0 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.253+0000 7fcadda38640 10 monclient: get_auth_request con 0x7fcabc06dc00 auth_method 0 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.253+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fcac8084ab0 con 0x7fcad80af8e0 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.253+0000 7fcaceffd640 10 client.4573.objecter ms_dispatch 0x7fcad8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.253+0000 7fcaceffd640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.253+0000 7fcaceffd640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.253+0000 7fcaceffd640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-29T11:29:10.246 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.253+0000 7fcaceffd640 10 client.4573.objecter ms_handle_connect 0x7fcabc06dc00 2024-03-29T11:29:10.248 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.258+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcac8050390 con 0x7fcad80af8e0 2024-03-29T11:29:10.248 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.258+0000 7fcaceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:10.248 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.258+0000 7fcaceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:10.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.608+0000 7fcae049d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:10.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.608+0000 7fcae049d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:10.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.608+0000 7fcae049d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:29:10.599 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.608+0000 7fcae049d640 1 -- 172.21.15.59:0/3036681906 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcad8072330 con 0x7fcad80af8e0 2024-03-29T11:29:10.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.610+0000 7fcaceffd640 1 -- 172.21.15.59:0/3036681906 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fcac80522c0 con 0x7fcad80af8e0 2024-03-29T11:29:10.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.610+0000 7fcaceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:10.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.610+0000 7fcaceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:10.601 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:29:10.601 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:10.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 10 librados: watch_flush enter 2024-03-29T11:29:10.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 10 librados: watch_flush exit 2024-03-29T11:29:10.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-29T11:29:10.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 10 client.4573.objecter successfully canceled tick 2024-03-29T11:29:10.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 1 -- 172.21.15.59:0/3036681906 >> v1:172.21.15.172:6816/2420742773 conn(0x7fcabc06dc00 legacy=0x7fcabc0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 10 monclient: shutdown 2024-03-29T11:29:10.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:10.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 1 -- 172.21.15.59:0/3036681906 >> v1:172.21.15.172:6789/0 conn(0x7fcad80af8e0 legacy=0x7fcad81d3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 1 -- 172.21.15.59:0/3036681906 shutdown_connections 2024-03-29T11:29:10.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 1 -- 172.21.15.59:0/3036681906 >> 172.21.15.59:0/3036681906 conn(0x7fcad806b660 msgr2=0x7fcad806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:10.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 1 -- 172.21.15.59:0/3036681906 shutdown_connections 2024-03-29T11:29:10.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 1 -- 172.21.15.59:0/3036681906 wait complete. 2024-03-29T11:29:10.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:10.614+0000 7fcae049d640 1 librados: shutdown 2024-03-29T11:29:10.623 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:17.430 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:17.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.597+0000 7f046c4d3640 10 monclient: get_monmap_and_config 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.597+0000 7f046c4d3640 10 monclient: build_initial_monmap 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.597+0000 7f046c4d3640 10 monclient: monmap: 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:29:17.598116+0000 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:29:17.598116+0000 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:29:17.593 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.598+0000 7f046c4d3640 1 Processor -- start 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.598+0000 7f046c4d3640 1 -- start start 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.598+0000 7f046c4d3640 10 monclient: init 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.599+0000 7f046c4d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046c4d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046c4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f04641457f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046c4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f046413f650 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046c4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f04640a9d70 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046c4d3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640aaef0 con 0x7f046413f650 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046c4d3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0464145bd0 con 0x7f04640a9d70 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046c4d3640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0464145eb0 con 0x7f04641457f0 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046c4d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046c4d3640 10 monclient(hunting): authenticate will time out at 1407.714131s 2024-03-29T11:29:17.594 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046aa70640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f04640a9d70 0x7f046414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:44210/0 (socket says 172.21.15.59:44210) 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046aa70640 1 -- 172.21.15.59:0/3778396802 learned_addr learned my addr 172.21.15.59:0/3778396802 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046926d640 1 -- 172.21.15.59:0/3778396802 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970408010 0 0) 0x7f0464145bd0 con 0x7f04640a9d70 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046926d640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046926d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046926d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046926d640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.600+0000 7f046926d640 1 -- 172.21.15.59:0/3778396802 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0450003520 con 0x7f04640a9d70 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 1 -- 172.21.15.59:0/3778396802 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492876019 0 0) 0x7f04640aaef0 con 0x7f046413f650 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 1 -- 172.21.15.59:0/3778396802 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0464145bd0 con 0x7f046413f650 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 1 -- 172.21.15.59:0/3778396802 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 703586228 0 0) 0x7f0450003520 con 0x7f04640a9d70 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 1 -- 172.21.15.59:0/3778396802 >> v1:172.21.15.59:6790/0 conn(0x7f04641457f0 legacy=0x7f046413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 1 -- 172.21.15.59:0/3778396802 >> v1:172.21.15.59:6789/0 conn(0x7f046413f650 legacy=0x7f04640b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 10 monclient: _finish_hunting 0 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 1 monclient: found mon.noname-c 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 1 -- 172.21.15.59:0/3778396802 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0464146340 con 0x7f04640a9d70 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 10 monclient: _finish_auth 0 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 10 monclient: _check_auth_tickets 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 1 -- 172.21.15.59:0/3778396802 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f045c001940 con 0x7f04640a9d70 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046926d640 10 monclient: dump: 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:17.596 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046c4d3640 5 monclient: authenticate success, global_id 4576 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.601+0000 7f046c4d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.602+0000 7f046926d640 1 -- 172.21.15.59:0/3778396802 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f045c001b60 con 0x7f04640a9d70 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.602+0000 7f046926d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.602+0000 7f046c4d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:17.597 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.602+0000 7f046c4d3640 4 set_mon_vals no callback set 2024-03-29T11:29:17.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.602+0000 7f046c4d3640 10 monclient: shutdown 2024-03-29T11:29:17.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.602+0000 7f046c4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:17.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.602+0000 7f046c4d3640 1 -- 172.21.15.59:0/3778396802 >> v1:172.21.15.172:6789/0 conn(0x7f04640a9d70 legacy=0x7f046414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.603+0000 7f046c4d3640 1 -- 172.21.15.59:0/3778396802 shutdown_connections 2024-03-29T11:29:17.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.603+0000 7f046c4d3640 1 -- 172.21.15.59:0/3778396802 >> 172.21.15.59:0/3778396802 conn(0x7f0464066a00 msgr2=0x7f0464066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:17.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.603+0000 7f046c4d3640 1 -- 172.21.15.59:0/3778396802 shutdown_connections 2024-03-29T11:29:17.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.603+0000 7f046c4d3640 1 -- 172.21.15.59:0/3778396802 wait complete. 2024-03-29T11:29:17.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.604+0000 7f046c4d3640 10 monclient: build_initial_monmap 2024-03-29T11:29:17.601 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.605+0000 7f046c4d3640 10 monclient: monmap: 2024-03-29T11:29:17.601 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:17.601 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.605+0000 7f046c4d3640 1 librados: starting msgr at 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.605+0000 7f046c4d3640 1 librados: starting objecter 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.605+0000 7f046c4d3640 1 Processor -- start 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.606+0000 7f046c4d3640 1 -- start start 2024-03-29T11:29:17.602 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.606+0000 7f046c4d3640 1 librados: setting wanted keys 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.606+0000 7f046c4d3640 1 librados: calling monclient init 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.606+0000 7f046c4d3640 10 monclient: init 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046c4d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046c4d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046c4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f04640a9d70 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046c4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f046413f650 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046c4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f04641457f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046c4d3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04641db190 con 0x7f046413f650 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046c4d3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04641d7cf0 con 0x7f04640a9d70 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046c4d3640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640b0990 con 0x7f04641457f0 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046c4d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046c4d3640 10 monclient(hunting): authenticate will time out at 1407.720812s 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046aa70640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f04641457f0 0x7f04640b0280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:51242/0 (socket says 172.21.15.59:51242) 2024-03-29T11:29:17.603 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f046aa70640 1 -- 172.21.15.59:0/3627265675 learned_addr learned my addr 172.21.15.59:0/3627265675 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f045affd640 10 client.?.objecter ms_handle_connect 0x7f04640a9d70 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f045affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.607+0000 7f045affd640 10 client.?.objecter ms_handle_connect 0x7f046413f650 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 10 client.?.objecter ms_handle_connect 0x7f04641457f0 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725552251 0 0) 0x7f04641d7cf0 con 0x7f04640a9d70 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0448004eb0 con 0x7f04640a9d70 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738420439 0 0) 0x7f04640b0990 con 0x7f04641457f0 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.604 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.608+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04641d7cf0 con 0x7f04641457f0 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1352705071 0 0) 0x7f0448004eb0 con 0x7f04640a9d70 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04640b0990 con 0x7f04640a9d70 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0454001890 con 0x7f04640a9d70 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 10 monclient(hunting): dump: 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:17.605 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185712819 0 0) 0x7f04641db190 con 0x7f046413f650 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0448004eb0 con 0x7f046413f650 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1009283815 0 0) 0x7f04641d7cf0 con 0x7f04641457f0 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04641db190 con 0x7f04641457f0 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f045c0012d0 con 0x7f04641457f0 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.606 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.609+0000 7f045affd640 10 monclient(hunting): dump: 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 272900881 0 0) 0x7f04640b0990 con 0x7f04640a9d70 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 >> v1:172.21.15.59:6790/0 conn(0x7f04641457f0 legacy=0x7f04640b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.607 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 >> v1:172.21.15.59:6789/0 conn(0x7f046413f650 legacy=0x7f04640afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f045affd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f045affd640 1 monclient: found mon.b 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f045affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f045affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04640b0d40 con 0x7f04640a9d70 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f045affd640 10 monclient: _finish_auth 0 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f045affd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f045affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f046c4d3640 5 monclient: authenticate success, global_id 4579 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f046c4d3640 10 monclient: _renew_subs 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f046c4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f046c4d3640 1 -- 172.21.15.59:0/3627265675 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04640b0ff0 con 0x7f04640a9d70 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f046c4d3640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f046c4d3640 10 monclient: _renew_subs 2024-03-29T11:29:17.608 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f046c4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:29:17.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f046c4d3640 1 -- 172.21.15.59:0/3627265675 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04641424c0 con 0x7f04640a9d70 2024-03-29T11:29:17.609 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.610+0000 7f046c4d3640 1 librados: init done 2024-03-29T11:29:17.610 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.611+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04540035f0 con 0x7f04640a9d70 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.611+0000 7f045affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.611+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f0454001d80 con 0x7f04640a9d70 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.611+0000 7f045affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.611+0000 7f045affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.611+0000 7f045affd640 10 monclient: dump: 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:17.611 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.611+0000 7f045bfff640 4 set_mon_vals no callback set 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.611+0000 7f046c4d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.612+0000 7f046c4d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.612+0000 7f046c4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.612+0000 7f046c4d3640 1 -- 172.21.15.59:0/3627265675 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044c005080 con 0x7f04640a9d70 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.613+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f0454003a00 con 0x7f04640a9d70 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.614+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7f04540837b0 con 0x7f04640a9d70 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.614+0000 7f045affd640 10 client.4579.objecter ms_dispatch 0x7f0464000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.614+0000 7f045affd640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.614+0000 7f045affd640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.614+0000 7f045affd640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.614+0000 7f0469a6e640 10 monclient: get_auth_request con 0x7f044806e050 auth_method 0 2024-03-29T11:29:17.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.615+0000 7f045affd640 10 client.4579.objecter ms_handle_connect 0x7f044806e050 2024-03-29T11:29:17.613 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.621+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f045404f090 con 0x7f04640a9d70 2024-03-29T11:29:17.613 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.621+0000 7f045affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:17.613 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:17.621+0000 7f045affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:18.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.033+0000 7f046c4d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:18.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.033+0000 7f046c4d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:18.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.033+0000 7f046c4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:29:18.024 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.033+0000 7f046c4d3640 1 -- 172.21.15.59:0/3627265675 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f044c005330 con 0x7f04640a9d70 2024-03-29T11:29:18.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.034+0000 7f045affd640 1 -- 172.21.15.59:0/3627265675 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7f0454050fc0 con 0x7f04640a9d70 2024-03-29T11:29:18.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.034+0000 7f045affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:18.025 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.034+0000 7f045affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:18.026 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:29:18.026 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:18.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.038+0000 7f046c4d3640 10 librados: watch_flush enter 2024-03-29T11:29:18.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.038+0000 7f046c4d3640 10 librados: watch_flush exit 2024-03-29T11:29:18.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.038+0000 7f046c4d3640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-29T11:29:18.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.038+0000 7f046c4d3640 10 client.4579.objecter successfully canceled tick 2024-03-29T11:29:18.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.039+0000 7f046c4d3640 1 -- 172.21.15.59:0/3627265675 >> v1:172.21.15.172:6816/2420742773 conn(0x7f044806e050 legacy=0x7f0448070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.039+0000 7f046c4d3640 10 monclient: shutdown 2024-03-29T11:29:18.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.039+0000 7f046c4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:18.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.039+0000 7f046c4d3640 1 -- 172.21.15.59:0/3627265675 >> v1:172.21.15.172:6789/0 conn(0x7f04640a9d70 legacy=0x7f04641d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.039+0000 7f046c4d3640 1 -- 172.21.15.59:0/3627265675 shutdown_connections 2024-03-29T11:29:18.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.039+0000 7f046c4d3640 1 -- 172.21.15.59:0/3627265675 >> 172.21.15.59:0/3627265675 conn(0x7f0464066a00 msgr2=0x7f046413ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:18.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.039+0000 7f046c4d3640 1 -- 172.21.15.59:0/3627265675 shutdown_connections 2024-03-29T11:29:18.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.039+0000 7f046c4d3640 1 -- 172.21.15.59:0/3627265675 wait complete. 2024-03-29T11:29:18.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:18.039+0000 7f046c4d3640 1 librados: shutdown 2024-03-29T11:29:18.046 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:24.856 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:25.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.016+0000 7fca429f6640 10 monclient: get_monmap_and_config 2024-03-29T11:29:25.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.016+0000 7fca429f6640 10 monclient: build_initial_monmap 2024-03-29T11:29:25.008 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.016+0000 7fca429f6640 10 monclient: monmap: 2024-03-29T11:29:25.008 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:25.008 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:25.009 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:29:25.017575+0000 2024-03-29T11:29:25.009 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:29:25.017575+0000 2024-03-29T11:29:25.009 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:25.009 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:25.009 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:25.009 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:29:25.009 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:29:25.009 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:25.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.018+0000 7fca429f6640 1 Processor -- start 2024-03-29T11:29:25.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.018+0000 7fca429f6640 1 -- start start 2024-03-29T11:29:25.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.018+0000 7fca429f6640 10 monclient: init 2024-03-29T11:29:25.011 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.018+0000 7fca429f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:25.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.018+0000 7fca429f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:25.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.018+0000 7fca429f6640 10 monclient(hunting): picked mon.noname-a con 0x7fca3c0b4100 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:25.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.019+0000 7fca429f6640 10 monclient(hunting): picked mon.noname-c con 0x7fca3c13ee90 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:25.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.019+0000 7fca429f6640 10 monclient(hunting): picked mon.noname-b con 0x7fca3c0af900 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:25.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.019+0000 7fca429f6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c0afd10 con 0x7fca3c0b4100 2024-03-29T11:29:25.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.019+0000 7fca429f6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c0af6f0 con 0x7fca3c13ee90 2024-03-29T11:29:25.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.019+0000 7fca429f6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c0aaf60 con 0x7fca3c0af900 2024-03-29T11:29:25.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.019+0000 7fca429f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:25.012 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.019+0000 7fca429f6640 10 monclient(hunting): authenticate will time out at 1415.132747s 2024-03-29T11:29:25.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.022+0000 7fca3bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fca3c0b4100 0x7fca3c0a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46531/0 (socket says 172.21.15.59:46531) 2024-03-29T11:29:25.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.022+0000 7fca3bfff640 1 -- 172.21.15.59:0/2476888700 learned_addr learned my addr 172.21.15.59:0/2476888700 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:25.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.022+0000 7fca3affd640 1 -- 172.21.15.59:0/2476888700 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782122569 0 0) 0x7fca3c0afd10 con 0x7fca3c0b4100 2024-03-29T11:29:25.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.022+0000 7fca3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:25.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.022+0000 7fca3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:25.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.022+0000 7fca3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:25.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.022+0000 7fca3affd640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:29:25.014 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.022+0000 7fca3affd640 1 -- 172.21.15.59:0/2476888700 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca24003520 con 0x7fca3c0b4100 2024-03-29T11:29:25.017 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 1 -- 172.21.15.59:0/2476888700 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3298353790 0 0) 0x7fca24003520 con 0x7fca3c0b4100 2024-03-29T11:29:25.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 1 -- 172.21.15.59:0/2476888700 >> v1:172.21.15.59:6790/0 conn(0x7fca3c0af900 legacy=0x7fca3c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:25.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 1 -- 172.21.15.59:0/2476888700 >> v1:172.21.15.172:6789/0 conn(0x7fca3c13ee90 legacy=0x7fca3c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:25.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 1 monclient: found mon.noname-a 2024-03-29T11:29:25.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:25.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:29:25.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 1 -- 172.21.15.59:0/2476888700 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c13e970 con 0x7fca3c0b4100 2024-03-29T11:29:25.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 10 monclient: _finish_auth 0 2024-03-29T11:29:25.018 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca429f6640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca429f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 1 -- 172.21.15.59:0/2476888700 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fca34001a10 con 0x7fca3c0b4100 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 10 monclient: dump: 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:25.019 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:25.020 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:25.020 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:25.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca429f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:25.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 1 -- 172.21.15.59:0/2476888700 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca34001ca0 con 0x7fca3c0b4100 2024-03-29T11:29:25.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:25.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca429f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:25.020 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.026+0000 7fca429f6640 4 set_mon_vals no callback set 2024-03-29T11:29:25.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.027+0000 7fca429f6640 10 monclient: shutdown 2024-03-29T11:29:25.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.027+0000 7fca429f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:25.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.027+0000 7fca429f6640 1 -- 172.21.15.59:0/2476888700 >> v1:172.21.15.59:6789/0 conn(0x7fca3c0b4100 legacy=0x7fca3c0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.028+0000 7fca429f6640 1 -- 172.21.15.59:0/2476888700 shutdown_connections 2024-03-29T11:29:25.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.028+0000 7fca429f6640 1 -- 172.21.15.59:0/2476888700 >> 172.21.15.59:0/2476888700 conn(0x7fca3c0b35e0 msgr2=0x7fca3c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:25.021 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.028+0000 7fca429f6640 1 -- 172.21.15.59:0/2476888700 shutdown_connections 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.028+0000 7fca429f6640 1 -- 172.21.15.59:0/2476888700 wait complete. 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.029+0000 7fca429f6640 10 monclient: build_initial_monmap 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.029+0000 7fca429f6640 10 monclient: monmap: 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:25.022 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.030+0000 7fca429f6640 1 librados: starting msgr at 2024-03-29T11:29:25.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.030+0000 7fca429f6640 1 librados: starting objecter 2024-03-29T11:29:25.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.030+0000 7fca429f6640 1 Processor -- start 2024-03-29T11:29:25.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 1 -- start start 2024-03-29T11:29:25.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 1 librados: setting wanted keys 2024-03-29T11:29:25.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 1 librados: calling monclient init 2024-03-29T11:29:25.023 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 10 monclient: init 2024-03-29T11:29:25.026 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:25.026 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:25.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 10 monclient(hunting): picked mon.noname-a con 0x7fca3c0af900 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:25.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 10 monclient(hunting): picked mon.noname-c con 0x7fca3c0b4100 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:25.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 10 monclient(hunting): picked mon.noname-b con 0x7fca3c13ee90 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:25.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c13f6f0 con 0x7fca3c0af900 2024-03-29T11:29:25.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c140820 con 0x7fca3c13ee90 2024-03-29T11:29:25.027 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c131490 con 0x7fca3c0b4100 2024-03-29T11:29:25.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:25.028 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.032+0000 7fca429f6640 10 monclient(hunting): authenticate will time out at 1415.146662s 2024-03-29T11:29:25.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.036+0000 7fca3bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fca3c0af900 0x7fca3c1e7640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46767/0 (socket says 172.21.15.59:46767) 2024-03-29T11:29:25.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.036+0000 7fca3bfff640 1 -- 172.21.15.59:0/3923562799 learned_addr learned my addr 172.21.15.59:0/3923562799 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:25.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.036+0000 7fca38ff9640 10 client.?.objecter ms_handle_connect 0x7fca3c0af900 2024-03-29T11:29:25.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.036+0000 7fca38ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:25.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.037+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670772257 0 0) 0x7fca3c13f6f0 con 0x7fca3c0af900 2024-03-29T11:29:25.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.037+0000 7fca38ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:25.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.037+0000 7fca38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:25.030 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.037+0000 7fca38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.037+0000 7fca38ff9640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.037+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca28004eb0 con 0x7fca3c0af900 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.038+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1566539069 0 0) 0x7fca28004eb0 con 0x7fca3c0af900 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.038+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca3c13f6f0 con 0x7fca3c0af900 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.038+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fca34004e60 con 0x7fca3c0af900 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.038+0000 7fca38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.038+0000 7fca38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.038+0000 7fca38ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:25.031 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:25.032 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:25.032 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:25.032 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:25.032 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:25.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.038+0000 7fca38ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:25.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.038+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2783746867 0 0) 0x7fca3c13f6f0 con 0x7fca3c0af900 2024-03-29T11:29:25.032 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.038+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 >> v1:172.21.15.59:6790/0 conn(0x7fca3c0b4100 legacy=0x7fca3c1e7d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:25.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 >> v1:172.21.15.172:6789/0 conn(0x7fca3c13ee90 legacy=0x7fca3c1eec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:25.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 1 monclient: found mon.a 2024-03-29T11:29:25.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:25.033 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:25.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c12ebb0 con 0x7fca3c0af900 2024-03-29T11:29:25.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:25.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:25.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:25.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca34005270 con 0x7fca3c0af900 2024-03-29T11:29:25.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:25.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fca340056c0 con 0x7fca3c0af900 2024-03-29T11:29:25.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:25.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:25.034 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca38ff9640 10 monclient: dump: 2024-03-29T11:29:25.034 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:25.035 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:25.035 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:25.035 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:25.035 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:25.035 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:25.035 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:25.035 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:25.035 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:25.035 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:25.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca39ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:25.035 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca429f6640 5 monclient: authenticate success, global_id 4578 2024-03-29T11:29:25.037 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca429f6640 10 monclient: _renew_subs 2024-03-29T11:29:25.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca429f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:25.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca429f6640 1 -- 172.21.15.59:0/3923562799 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3c12ee60 con 0x7fca3c0af900 2024-03-29T11:29:25.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca429f6640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:25.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca429f6640 10 monclient: _renew_subs 2024-03-29T11:29:25.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca429f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:25.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca429f6640 1 -- 172.21.15.59:0/3923562799 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c132240 con 0x7fca3c0af900 2024-03-29T11:29:25.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.041+0000 7fca429f6640 1 librados: init done 2024-03-29T11:29:25.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.045+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fca34001e90 con 0x7fca3c0af900 2024-03-29T11:29:25.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.046+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fca34085180 con 0x7fca3c0af900 2024-03-29T11:29:25.038 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.046+0000 7fca38ff9640 10 client.4578.objecter ms_dispatch 0x7fca3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:25.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.046+0000 7fca38ff9640 3 client.4578.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:25.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.046+0000 7fca38ff9640 3 client.4578.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:25.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.046+0000 7fca38ff9640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-29T11:29:25.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.046+0000 7fca3b7fe640 10 monclient: get_auth_request con 0x7fca2806dcd0 auth_method 0 2024-03-29T11:29:25.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.047+0000 7fca429f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:25.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.047+0000 7fca429f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:25.039 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.047+0000 7fca429f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:25.040 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.047+0000 7fca429f6640 1 -- 172.21.15.59:0/3923562799 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3c14cda0 con 0x7fca3c0af900 2024-03-29T11:29:25.043 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.050+0000 7fca38ff9640 10 client.4578.objecter ms_handle_connect 0x7fca2806dcd0 2024-03-29T11:29:25.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.057+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca34050a60 con 0x7fca3c0af900 2024-03-29T11:29:25.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.057+0000 7fca38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:25.048 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.057+0000 7fca38ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:25.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.440+0000 7fca429f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:25.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.440+0000 7fca429f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:25.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.440+0000 7fca429f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:25.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.440+0000 7fca429f6640 1 -- 172.21.15.59:0/3923562799 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca3c075f10 con 0x7fca3c0af900 2024-03-29T11:29:25.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.441+0000 7fca38ff9640 1 -- 172.21.15.59:0/3923562799 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fca34052990 con 0x7fca3c0af900 2024-03-29T11:29:25.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.441+0000 7fca38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:25.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.441+0000 7fca38ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:25.433 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:29:25.433 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:25.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 10 librados: watch_flush enter 2024-03-29T11:29:25.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 10 librados: watch_flush exit 2024-03-29T11:29:25.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-29T11:29:25.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 10 client.4578.objecter successfully canceled tick 2024-03-29T11:29:25.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 1 -- 172.21.15.59:0/3923562799 >> v1:172.21.15.172:6816/2420742773 conn(0x7fca2806dcd0 legacy=0x7fca28070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 10 monclient: shutdown 2024-03-29T11:29:25.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:25.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 1 -- 172.21.15.59:0/3923562799 >> v1:172.21.15.59:6789/0 conn(0x7fca3c0af900 legacy=0x7fca3c1e7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 1 -- 172.21.15.59:0/3923562799 shutdown_connections 2024-03-29T11:29:25.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 1 -- 172.21.15.59:0/3923562799 >> 172.21.15.59:0/3923562799 conn(0x7fca3c0b35e0 msgr2=0x7fca3c14d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:25.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 1 -- 172.21.15.59:0/3923562799 shutdown_connections 2024-03-29T11:29:25.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.445+0000 7fca429f6640 1 -- 172.21.15.59:0/3923562799 wait complete. 2024-03-29T11:29:25.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:25.446+0000 7fca429f6640 1 librados: shutdown 2024-03-29T11:29:25.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:32.257 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:32.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.417+0000 7fb4e948d640 10 monclient: get_monmap_and_config 2024-03-29T11:29:32.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.417+0000 7fb4e948d640 10 monclient: build_initial_monmap 2024-03-29T11:29:32.408 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.417+0000 7fb4e948d640 10 monclient: monmap: 2024-03-29T11:29:32.408 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:32.409 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:32.409 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:29:32.417855+0000 2024-03-29T11:29:32.409 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:29:32.417855+0000 2024-03-29T11:29:32.409 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:32.409 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:32.409 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:32.409 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:29:32.409 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:29:32.409 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:32.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.418+0000 7fb4e948d640 1 Processor -- start 2024-03-29T11:29:32.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.418+0000 7fb4e948d640 1 -- start start 2024-03-29T11:29:32.410 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.418+0000 7fb4e948d640 10 monclient: init 2024-03-29T11:29:32.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e948d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:32.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e948d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:32.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e948d640 10 monclient(hunting): picked mon.noname-c con 0x7fb4e40b4920 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:32.412 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e948d640 10 monclient(hunting): picked mon.noname-a con 0x7fb4e40b1790 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:32.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e948d640 10 monclient(hunting): picked mon.noname-b con 0x7fb4e4146790 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:32.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e948d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e40b4070 con 0x7fb4e40b1790 2024-03-29T11:29:32.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e948d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e406fb80 con 0x7fb4e40b4920 2024-03-29T11:29:32.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e948d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e40b42c0 con 0x7fb4e4146790 2024-03-29T11:29:32.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e948d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:32.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e948d640 10 monclient(hunting): authenticate will time out at 1422.533103s 2024-03-29T11:29:32.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e27fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb4e40b1790 0x7fb4e40b1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:33804/0 (socket says 172.21.15.59:33804) 2024-03-29T11:29:32.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.419+0000 7fb4e27fc640 1 -- 172.21.15.59:0/3967829637 learned_addr learned my addr 172.21.15.59:0/3967829637 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:32.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.420+0000 7fb4e1ffb640 1 -- 172.21.15.59:0/3967829637 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066575513 0 0) 0x7fb4e40b4070 con 0x7fb4e40b1790 2024-03-29T11:29:32.413 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.420+0000 7fb4e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.420+0000 7fb4e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.420+0000 7fb4e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.420+0000 7fb4e1ffb640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:29:32.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.420+0000 7fb4e1ffb640 1 -- 172.21.15.59:0/3967829637 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4cc003520 con 0x7fb4e40b1790 2024-03-29T11:29:32.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.420+0000 7fb4e1ffb640 1 -- 172.21.15.59:0/3967829637 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2510714734 0 0) 0x7fb4cc003520 con 0x7fb4e40b1790 2024-03-29T11:29:32.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.420+0000 7fb4e1ffb640 1 -- 172.21.15.59:0/3967829637 >> v1:172.21.15.59:6790/0 conn(0x7fb4e4146790 legacy=0x7fb4e4146c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:32.414 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.420+0000 7fb4e1ffb640 1 -- 172.21.15.59:0/3967829637 >> v1:172.21.15.172:6789/0 conn(0x7fb4e40b4920 legacy=0x7fb4e406e510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:32.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:32.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:29:32.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:32.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:29:32.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 1 -- 172.21.15.59:0/3967829637 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e40b22c0 con 0x7fb4e40b1790 2024-03-29T11:29:32.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:32.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:32.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:32.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 1 -- 172.21.15.59:0/3967829637 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb4d8001690 con 0x7fb4e40b1790 2024-03-29T11:29:32.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:32.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.422+0000 7fb4e1ffb640 10 monclient: dump: 2024-03-29T11:29:32.417 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:32.417 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:32.417 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:32.418 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:32.418 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.418 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:32.418 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:32.418 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:32.418 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:32.418 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:32.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.423+0000 7fb4e948d640 5 monclient: authenticate success, global_id 4581 2024-03-29T11:29:32.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.423+0000 7fb4e948d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:32.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.424+0000 7fb4e1ffb640 1 -- 172.21.15.59:0/3967829637 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4d8001920 con 0x7fb4e40b1790 2024-03-29T11:29:32.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.424+0000 7fb4e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.424+0000 7fb4e1ffb640 1 -- 172.21.15.59:0/3967829637 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb4d8001cb0 con 0x7fb4e40b1790 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.424+0000 7fb4e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.425+0000 7fb4e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.425+0000 7fb4e1ffb640 10 monclient: dump: 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:32.419 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:32.420 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:32.420 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:32.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.425+0000 7fb4e948d640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:32.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.425+0000 7fb4e948d640 4 set_mon_vals no callback set 2024-03-29T11:29:32.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.426+0000 7fb4e948d640 10 monclient: shutdown 2024-03-29T11:29:32.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.426+0000 7fb4e948d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:32.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.426+0000 7fb4e948d640 1 -- 172.21.15.59:0/3967829637 >> v1:172.21.15.59:6789/0 conn(0x7fb4e40b1790 legacy=0x7fb4e40b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.429+0000 7fb4e948d640 1 -- 172.21.15.59:0/3967829637 shutdown_connections 2024-03-29T11:29:32.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.429+0000 7fb4e948d640 1 -- 172.21.15.59:0/3967829637 >> 172.21.15.59:0/3967829637 conn(0x7fb4e4066a00 msgr2=0x7fb4e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:32.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.429+0000 7fb4e948d640 1 -- 172.21.15.59:0/3967829637 shutdown_connections 2024-03-29T11:29:32.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.429+0000 7fb4e948d640 1 -- 172.21.15.59:0/3967829637 wait complete. 2024-03-29T11:29:32.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.430+0000 7fb4e948d640 10 monclient: build_initial_monmap 2024-03-29T11:29:32.422 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.430+0000 7fb4e948d640 10 monclient: monmap: 2024-03-29T11:29:32.422 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:32.422 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:32.422 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:29:32.422 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:29:32.422 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:32.422 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:32.423 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:32.423 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:29:32.423 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:29:32.423 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:32.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.431+0000 7fb4e948d640 1 librados: starting msgr at 2024-03-29T11:29:32.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.431+0000 7fb4e948d640 1 librados: starting objecter 2024-03-29T11:29:32.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.431+0000 7fb4e948d640 1 Processor -- start 2024-03-29T11:29:32.423 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.431+0000 7fb4e948d640 1 -- start start 2024-03-29T11:29:32.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.431+0000 7fb4e948d640 1 librados: setting wanted keys 2024-03-29T11:29:32.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.431+0000 7fb4e948d640 1 librados: calling monclient init 2024-03-29T11:29:32.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.431+0000 7fb4e948d640 10 monclient: init 2024-03-29T11:29:32.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.432+0000 7fb4e948d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:32.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.432+0000 7fb4e948d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:32.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.432+0000 7fb4e948d640 10 monclient(hunting): picked mon.noname-c con 0x7fb4e40b1790 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:32.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.432+0000 7fb4e948d640 10 monclient(hunting): picked mon.noname-b con 0x7fb4e40b4920 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:32.426 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.432+0000 7fb4e948d640 10 monclient(hunting): picked mon.noname-a con 0x7fb4e4146790 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:32.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.432+0000 7fb4e948d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e41418b0 con 0x7fb4e4146790 2024-03-29T11:29:32.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.432+0000 7fb4e948d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e4141b90 con 0x7fb4e40b4920 2024-03-29T11:29:32.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.432+0000 7fb4e948d640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e4141e70 con 0x7fb4e40b1790 2024-03-29T11:29:32.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.432+0000 7fb4e948d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:32.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.432+0000 7fb4e948d640 10 monclient(hunting): authenticate will time out at 1422.546126s 2024-03-29T11:29:32.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.434+0000 7fb4e2ffd640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fb4e40b1790 0x7fb4e4139800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:33129/0 (socket says 172.21.15.59:33129) 2024-03-29T11:29:32.427 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.434+0000 7fb4e2ffd640 1 -- 172.21.15.59:0/2557012177 learned_addr learned my addr 172.21.15.59:0/2557012177 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:32.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.435+0000 7fb4cb7fe640 10 client.?.objecter ms_handle_connect 0x7fb4e40b1790 2024-03-29T11:29:32.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.435+0000 7fb4cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:32.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.435+0000 7fb4cb7fe640 10 client.?.objecter ms_handle_connect 0x7fb4e4146790 2024-03-29T11:29:32.428 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.435+0000 7fb4cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:32.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.435+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121908584 0 0) 0x7fb4e4141e70 con 0x7fb4e40b1790 2024-03-29T11:29:32.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:29:32.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d0004eb0 con 0x7fb4e40b1790 2024-03-29T11:29:32.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900777034 0 0) 0x7fb4e41418b0 con 0x7fb4e4146790 2024-03-29T11:29:32.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:29:32.431 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e4141e70 con 0x7fb4e4146790 2024-03-29T11:29:32.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 10 client.?.objecter ms_handle_connect 0x7fb4e40b4920 2024-03-29T11:29:32.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.438+0000 7fb4cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:32.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1885874522 0 0) 0x7fb4e4141e70 con 0x7fb4e4146790 2024-03-29T11:29:32.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4e41418b0 con 0x7fb4e4146790 2024-03-29T11:29:32.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb4d4001650 con 0x7fb4e4146790 2024-03-29T11:29:32.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.432 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:32.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient(hunting): dump: 2024-03-29T11:29:32.437 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:32.437 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3709677723 0 0) 0x7fb4d0004eb0 con 0x7fb4e40b1790 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4e4141e70 con 0x7fb4e40b1790 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388325102 0 0) 0x7fb4e4141b90 con 0x7fb4e40b4920 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient(hunting): my global_id is 4585 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d0004eb0 con 0x7fb4e40b4920 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1116549106 0 0) 0x7fb4e41418b0 con 0x7fb4e4146790 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 >> v1:172.21.15.59:6790/0 conn(0x7fb4e40b1790 legacy=0x7fb4e4139800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 >> v1:172.21.15.172:6789/0 conn(0x7fb4e40b4920 legacy=0x7fb4e4137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 monclient: found mon.a 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e4139f10 con 0x7fb4e4146790 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:32.439 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4e948d640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4e948d640 10 monclient: _renew_subs 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4e948d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4e948d640 1 -- 172.21.15.59:0/2557012177 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e4137f20 con 0x7fb4e4146790 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4e948d640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4e948d640 10 monclient: _renew_subs 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4e948d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4e948d640 1 -- 172.21.15.59:0/2557012177 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e41430e0 con 0x7fb4e4146790 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.439+0000 7fb4e948d640 1 librados: init done 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.443+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4d4001b60 con 0x7fb4e4146790 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.443+0000 7fb4cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.443+0000 7fb4e17fa640 4 set_mon_vals no callback set 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.443+0000 7fb4e948d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.444+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fb4d4003600 con 0x7fb4e4146790 2024-03-29T11:29:32.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.444+0000 7fb4e948d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.444+0000 7fb4e948d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.444+0000 7fb4e948d640 1 -- 172.21.15.59:0/2557012177 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c4005080 con 0x7fb4e4146790 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.444+0000 7fb4cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.444+0000 7fb4cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.444+0000 7fb4cb7fe640 10 monclient: dump: 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:32.441 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:32.442 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.444+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fb4d40195d0 con 0x7fb4e4146790 2024-03-29T11:29:32.442 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.445+0000 7fb4e2ffd640 10 monclient: get_auth_request con 0x7fb4d006dd70 auth_method 0 2024-03-29T11:29:32.442 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.445+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fb4d4083550 con 0x7fb4e4146790 2024-03-29T11:29:32.442 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.445+0000 7fb4cb7fe640 10 client.4584.objecter ms_dispatch 0x7fb4e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:32.442 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.445+0000 7fb4cb7fe640 3 client.4584.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:32.442 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.445+0000 7fb4cb7fe640 3 client.4584.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:32.442 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.445+0000 7fb4cb7fe640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-29T11:29:32.442 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.448+0000 7fb4cb7fe640 10 client.4584.objecter ms_handle_connect 0x7fb4d006dd70 2024-03-29T11:29:32.443 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.452+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4d404eeb0 con 0x7fb4e4146790 2024-03-29T11:29:32.443 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.452+0000 7fb4cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:32.443 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.452+0000 7fb4cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:32.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.782+0000 7fb4e948d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.782+0000 7fb4e948d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.782+0000 7fb4e948d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:32.773 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.782+0000 7fb4e948d640 1 -- 172.21.15.59:0/2557012177 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4c4005330 con 0x7fb4e4146790 2024-03-29T11:29:32.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.783+0000 7fb4cb7fe640 1 -- 172.21.15.59:0/2557012177 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fb4d4050de0 con 0x7fb4e4146790 2024-03-29T11:29:32.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.783+0000 7fb4cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.774 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.783+0000 7fb4cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:32.774 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:29:32.774 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:32.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.786+0000 7fb4e948d640 10 librados: watch_flush enter 2024-03-29T11:29:32.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.786+0000 7fb4e948d640 10 librados: watch_flush exit 2024-03-29T11:29:32.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.786+0000 7fb4e948d640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-29T11:29:32.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.786+0000 7fb4e948d640 10 client.4584.objecter successfully canceled tick 2024-03-29T11:29:32.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.786+0000 7fb4e948d640 1 -- 172.21.15.59:0/2557012177 >> v1:172.21.15.172:6816/2420742773 conn(0x7fb4d006dd70 legacy=0x7fb4d0070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.786+0000 7fb4e948d640 10 monclient: shutdown 2024-03-29T11:29:32.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.786+0000 7fb4e948d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:32.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.786+0000 7fb4e948d640 1 -- 172.21.15.59:0/2557012177 >> v1:172.21.15.59:6789/0 conn(0x7fb4e4146790 legacy=0x7fb4e4137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.787+0000 7fb4e948d640 1 -- 172.21.15.59:0/2557012177 shutdown_connections 2024-03-29T11:29:32.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.787+0000 7fb4e948d640 1 -- 172.21.15.59:0/2557012177 >> 172.21.15.59:0/2557012177 conn(0x7fb4e4066a00 msgr2=0x7fb4e406e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:32.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.787+0000 7fb4e948d640 1 -- 172.21.15.59:0/2557012177 shutdown_connections 2024-03-29T11:29:32.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.787+0000 7fb4e948d640 1 -- 172.21.15.59:0/2557012177 wait complete. 2024-03-29T11:29:32.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:32.787+0000 7fb4e948d640 1 librados: shutdown 2024-03-29T11:29:32.794 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:39.602 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:39.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.748+0000 7fef27a6e640 10 monclient: get_monmap_and_config 2024-03-29T11:29:39.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.748+0000 7fef27a6e640 10 monclient: build_initial_monmap 2024-03-29T11:29:39.740 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.748+0000 7fef27a6e640 10 monclient: monmap: 2024-03-29T11:29:39.740 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:39.740 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:39.740 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:29:39.749475+0000 2024-03-29T11:29:39.740 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:29:39.749475+0000 2024-03-29T11:29:39.741 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:39.741 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:39.741 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:39.741 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:29:39.741 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:29:39.741 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:39.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.749+0000 7fef27a6e640 1 Processor -- start 2024-03-29T11:29:39.741 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.750+0000 7fef27a6e640 1 -- start start 2024-03-29T11:29:39.742 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.750+0000 7fef27a6e640 10 monclient: init 2024-03-29T11:29:39.744 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef27a6e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:39.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef27a6e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:39.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef27a6e640 10 monclient(hunting): picked mon.noname-c con 0x7fef201457f0 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:39.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef27a6e640 10 monclient(hunting): picked mon.noname-a con 0x7fef2013f650 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:39.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef27a6e640 10 monclient(hunting): picked mon.noname-b con 0x7fef200a9d70 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:39.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef27a6e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef200aaef0 con 0x7fef2013f650 2024-03-29T11:29:39.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef27a6e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef20145bd0 con 0x7fef201457f0 2024-03-29T11:29:39.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef27a6e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef20145eb0 con 0x7fef200a9d70 2024-03-29T11:29:39.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef27a6e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:39.745 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef27a6e640 10 monclient(hunting): authenticate will time out at 1429.865151s 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef2580a640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fef201457f0 0x7fef2013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:36456/0 (socket says 172.21.15.59:36456) 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef2580a640 1 -- 172.21.15.59:0/964970559 learned_addr learned my addr 172.21.15.59:0/964970559 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef24808640 1 -- 172.21.15.59:0/964970559 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846358027 0 0) 0x7fef20145bd0 con 0x7fef201457f0 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef24808640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef24808640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef24808640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.751+0000 7fef24808640 10 monclient(hunting): my global_id is 4588 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.752+0000 7fef24808640 1 -- 172.21.15.59:0/964970559 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef0c003520 con 0x7fef201457f0 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.752+0000 7fef24808640 1 -- 172.21.15.59:0/964970559 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341538975 0 0) 0x7fef20145eb0 con 0x7fef200a9d70 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.752+0000 7fef24808640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.752+0000 7fef24808640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.752+0000 7fef24808640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.746 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.752+0000 7fef24808640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.752+0000 7fef24808640 1 -- 172.21.15.59:0/964970559 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef20145bd0 con 0x7fef200a9d70 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.752+0000 7fef24808640 1 -- 172.21.15.59:0/964970559 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4007357783 0 0) 0x7fef0c003520 con 0x7fef201457f0 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.752+0000 7fef24808640 1 -- 172.21.15.59:0/964970559 >> v1:172.21.15.59:6790/0 conn(0x7fef200a9d70 legacy=0x7fef2014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.752+0000 7fef24808640 1 -- 172.21.15.59:0/964970559 >> v1:172.21.15.59:6789/0 conn(0x7fef2013f650 legacy=0x7fef200b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 10 monclient: _finish_hunting 0 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 1 monclient: found mon.noname-c 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 1 -- 172.21.15.59:0/964970559 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef20146340 con 0x7fef201457f0 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 10 monclient: _finish_auth 0 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 10 monclient: _check_auth_tickets 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 1 -- 172.21.15.59:0/964970559 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fef18001690 con 0x7fef201457f0 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.747 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef24808640 10 monclient: dump: 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef27a6e640 5 monclient: authenticate success, global_id 4588 2024-03-29T11:29:39.748 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.753+0000 7fef27a6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:39.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.755+0000 7fef24808640 1 -- 172.21.15.59:0/964970559 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef18001870 con 0x7fef201457f0 2024-03-29T11:29:39.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.755+0000 7fef24808640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:39.750 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.755+0000 7fef27a6e640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.755+0000 7fef27a6e640 4 set_mon_vals no callback set 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.755+0000 7fef24808640 1 -- 172.21.15.59:0/964970559 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fef18001cc0 con 0x7fef201457f0 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.755+0000 7fef24808640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.755+0000 7fef24808640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.755+0000 7fef24808640 10 monclient: dump: 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:39.751 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.756+0000 7fef27a6e640 10 monclient: shutdown 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.756+0000 7fef27a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.756+0000 7fef27a6e640 1 -- 172.21.15.59:0/964970559 >> v1:172.21.15.172:6789/0 conn(0x7fef201457f0 legacy=0x7fef2013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.756+0000 7fef27a6e640 1 -- 172.21.15.59:0/964970559 shutdown_connections 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.756+0000 7fef27a6e640 1 -- 172.21.15.59:0/964970559 >> 172.21.15.59:0/964970559 conn(0x7fef20066a00 msgr2=0x7fef20066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.756+0000 7fef27a6e640 1 -- 172.21.15.59:0/964970559 shutdown_connections 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.756+0000 7fef27a6e640 1 -- 172.21.15.59:0/964970559 wait complete. 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.757+0000 7fef27a6e640 10 monclient: build_initial_monmap 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.757+0000 7fef27a6e640 10 monclient: monmap: 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:39.752 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.757+0000 7fef27a6e640 1 librados: starting msgr at 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.757+0000 7fef27a6e640 1 librados: starting objecter 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.757+0000 7fef27a6e640 1 Processor -- start 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.757+0000 7fef27a6e640 1 -- start start 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.757+0000 7fef27a6e640 1 librados: setting wanted keys 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.757+0000 7fef27a6e640 1 librados: calling monclient init 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.757+0000 7fef27a6e640 10 monclient: init 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.758+0000 7fef27a6e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.758+0000 7fef27a6e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.758+0000 7fef27a6e640 10 monclient(hunting): picked mon.noname-a con 0x7fef200a9d70 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:39.753 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.758+0000 7fef27a6e640 10 monclient(hunting): picked mon.noname-b con 0x7fef2013f650 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:39.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.759+0000 7fef27a6e640 10 monclient(hunting): picked mon.noname-c con 0x7fef201457f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:39.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.759+0000 7fef27a6e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2013a830 con 0x7fef200a9d70 2024-03-29T11:29:39.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.759+0000 7fef27a6e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2013b410 con 0x7fef2013f650 2024-03-29T11:29:39.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.759+0000 7fef2600b640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7fef201457f0 0x7fef201d7010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:51028/0 (socket says 172.21.15.59:51028) 2024-03-29T11:29:39.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.759+0000 7fef2600b640 1 -- 172.21.15.59:0/4026747357 learned_addr learned my addr 172.21.15.59:0/4026747357 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:39.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.759+0000 7fef27a6e640 1 -- 172.21.15.59:0/4026747357 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2013b930 con 0x7fef201457f0 2024-03-29T11:29:39.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.759+0000 7fef27a6e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:39.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.759+0000 7fef167fc640 10 client.?.objecter ms_handle_connect 0x7fef201457f0 2024-03-29T11:29:39.754 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.759+0000 7fef167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:39.755 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.759+0000 7fef27a6e640 10 monclient(hunting): authenticate will time out at 1429.872993s 2024-03-29T11:29:39.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648372006 0 0) 0x7fef2013b930 con 0x7fef201457f0 2024-03-29T11:29:39.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:29:39.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef04004eb0 con 0x7fef201457f0 2024-03-29T11:29:39.757 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1391315272 0 0) 0x7fef04004eb0 con 0x7fef201457f0 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef2013b930 con 0x7fef201457f0 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fef10001a10 con 0x7fef201457f0 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 10 monclient(hunting): dump: 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:39.758 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3172575964 0 0) 0x7fef2013b930 con 0x7fef201457f0 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 >> v1:172.21.15.172:6789/0 conn(0x7fef2013f650 legacy=0x7fef201d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.760+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 >> v1:172.21.15.59:6789/0 conn(0x7fef200a9d70 legacy=0x7fef20139150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 1 monclient: found mon.c 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:29:39.759 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef201d4ae0 con 0x7fef201457f0 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef27a6e640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef27a6e640 10 monclient: _renew_subs 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef27a6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef27a6e640 1 -- 172.21.15.59:0/4026747357 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef20138a90 con 0x7fef201457f0 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef10001c50 con 0x7fef201457f0 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fef10003a40 con 0x7fef201457f0 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef167fc640 10 monclient: dump: 2024-03-29T11:29:39.760 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef17fff640 4 set_mon_vals no callback set 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef27a6e640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef27a6e640 10 monclient: _renew_subs 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef27a6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef27a6e640 1 -- 172.21.15.59:0/4026747357 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2013b110 con 0x7fef201457f0 2024-03-29T11:29:39.761 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.761+0000 7fef27a6e640 1 librados: init done 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.763+0000 7fef27a6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.763+0000 7fef27a6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.763+0000 7fef27a6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.763+0000 7fef27a6e640 1 -- 172.21.15.59:0/4026747357 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef20075f10 con 0x7fef201457f0 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.763+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fef10003ba0 con 0x7fef201457f0 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.764+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 132282051 0 0) 0x7fef10084b60 con 0x7fef201457f0 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.764+0000 7fef167fc640 10 client.4607.objecter ms_dispatch 0x7fef20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.764+0000 7fef167fc640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.764+0000 7fef167fc640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.764+0000 7fef167fc640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.765+0000 7fef2580a640 10 monclient: get_auth_request con 0x7fef0406dbb0 auth_method 0 2024-03-29T11:29:39.762 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.765+0000 7fef167fc640 10 client.4607.objecter ms_handle_connect 0x7fef0406dbb0 2024-03-29T11:29:39.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.770+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef10050440 con 0x7fef201457f0 2024-03-29T11:29:39.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.770+0000 7fef167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.763 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:39.770+0000 7fef167fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:40.067 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.076+0000 7fef27a6e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:40.068 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.076+0000 7fef27a6e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:40.068 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.076+0000 7fef27a6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:29:40.068 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.076+0000 7fef27a6e640 1 -- 172.21.15.59:0/4026747357 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef20139920 con 0x7fef201457f0 2024-03-29T11:29:40.068 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.077+0000 7fef167fc640 1 -- 172.21.15.59:0/4026747357 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3300676006) 0x7fef10052370 con 0x7fef201457f0 2024-03-29T11:29:40.068 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.077+0000 7fef167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:40.068 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.077+0000 7fef167fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:40.069 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:29:40.069 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:19:42.273959+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:42.273959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:40.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 10 librados: watch_flush enter 2024-03-29T11:29:40.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 10 librados: watch_flush exit 2024-03-29T11:29:40.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T11:29:40.071 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 10 client.4607.objecter successfully canceled tick 2024-03-29T11:29:40.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 1 -- 172.21.15.59:0/4026747357 >> v1:172.21.15.172:6816/2420742773 conn(0x7fef0406dbb0 legacy=0x7fef04070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 10 monclient: shutdown 2024-03-29T11:29:40.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:40.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 1 -- 172.21.15.59:0/4026747357 >> v1:172.21.15.59:6790/0 conn(0x7fef201457f0 legacy=0x7fef201d7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 1 -- 172.21.15.59:0/4026747357 shutdown_connections 2024-03-29T11:29:40.072 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 1 -- 172.21.15.59:0/4026747357 >> 172.21.15.59:0/4026747357 conn(0x7fef20066a00 msgr2=0x7fef200b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:40.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 1 -- 172.21.15.59:0/4026747357 shutdown_connections 2024-03-29T11:29:40.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 1 -- 172.21.15.59:0/4026747357 wait complete. 2024-03-29T11:29:40.073 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:40.080+0000 7fef27a6e640 1 librados: shutdown 2024-03-29T11:29:40.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:46.894 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:47.040 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.048+0000 7f1ea6026640 10 monclient: get_monmap_and_config 2024-03-29T11:29:47.040 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.048+0000 7f1ea6026640 10 monclient: build_initial_monmap 2024-03-29T11:29:47.040 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.048+0000 7f1ea6026640 10 monclient: monmap: 2024-03-29T11:29:47.040 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:47.040 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:47.040 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:29:47.049430+0000 2024-03-29T11:29:47.040 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:29:47.049430+0000 2024-03-29T11:29:47.040 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:47.041 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:47.041 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:47.041 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:29:47.041 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:29:47.041 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:47.041 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.050+0000 7f1ea6026640 1 Processor -- start 2024-03-29T11:29:47.042 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.050+0000 7f1ea6026640 1 -- start start 2024-03-29T11:29:47.042 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.050+0000 7f1ea6026640 10 monclient: init 2024-03-29T11:29:47.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.051+0000 7f1ea6026640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:47.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.051+0000 7f1ea6026640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:47.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.051+0000 7f1ea6026640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea00b4820 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:47.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.051+0000 7f1ea6026640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea00a9ce0 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:47.044 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.053+0000 7f1ea5024640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1ea00b4820 0x7f1ea013baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:52410/0 (socket says 172.21.15.59:52410) 2024-03-29T11:29:47.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.054+0000 7f1ea6026640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea00b3cd0 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:47.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.054+0000 7f1ea6026640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea00b4180 con 0x7f1ea00a9ce0 2024-03-29T11:29:47.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.054+0000 7f1ea6026640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea00aaf60 con 0x7f1ea00b4820 2024-03-29T11:29:47.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.054+0000 7f1ea6026640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea013c200 con 0x7f1ea00b3cd0 2024-03-29T11:29:47.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.054+0000 7f1ea6026640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:47.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.054+0000 7f1ea6026640 10 monclient(hunting): authenticate will time out at 1437.168361s 2024-03-29T11:29:47.046 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.054+0000 7f1ea5024640 1 -- 172.21.15.59:0/3777231755 learned_addr learned my addr 172.21.15.59:0/3777231755 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:47.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.055+0000 7f1e97fff640 1 -- 172.21.15.59:0/3777231755 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248798609 0 0) 0x7f1ea00aaf60 con 0x7f1ea00b4820 2024-03-29T11:29:47.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.055+0000 7f1e97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.055+0000 7f1e97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.055+0000 7f1e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.055+0000 7f1e97fff640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:29:47.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.055+0000 7f1e97fff640 1 -- 172.21.15.59:0/3777231755 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e8c003520 con 0x7f1ea00b4820 2024-03-29T11:29:47.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.056+0000 7f1e97fff640 1 -- 172.21.15.59:0/3777231755 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2879617181 0 0) 0x7f1e8c003520 con 0x7f1ea00b4820 2024-03-29T11:29:47.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.056+0000 7f1e97fff640 1 -- 172.21.15.59:0/3777231755 >> v1:172.21.15.59:6790/0 conn(0x7f1ea00b3cd0 legacy=0x7f1ea014d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:47.047 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.056+0000 7f1e97fff640 1 -- 172.21.15.59:0/3777231755 >> v1:172.21.15.59:6789/0 conn(0x7f1ea00a9ce0 legacy=0x7f1ea0145770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 1 monclient: found mon.noname-c 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 1 -- 172.21.15.59:0/3777231755 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea0145e80 con 0x7f1ea00b4820 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 1 -- 172.21.15.59:0/3777231755 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1e9c001690 con 0x7f1ea00b4820 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:47.050 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1e97fff640 10 monclient: dump: 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1ea6026640 5 monclient: authenticate success, global_id 4591 2024-03-29T11:29:47.051 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.057+0000 7f1ea6026640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:47.052 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.059+0000 7f1e97fff640 1 -- 172.21.15.59:0/3777231755 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e9c0018f0 con 0x7f1ea00b4820 2024-03-29T11:29:47.052 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.059+0000 7f1e97fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:47.052 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.059+0000 7f1e97fff640 1 -- 172.21.15.59:0/3777231755 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1e9c001d40 con 0x7f1ea00b4820 2024-03-29T11:29:47.052 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.060+0000 7f1e97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.052 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.060+0000 7f1e97fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:47.053 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.060+0000 7f1e97fff640 10 monclient: dump: 2024-03-29T11:29:47.053 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:47.053 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:47.053 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:47.053 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:47.053 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.053 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:47.053 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:47.053 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:47.053 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:47.054 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:47.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.060+0000 7f1ea6026640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:47.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.060+0000 7f1ea6026640 4 set_mon_vals no callback set 2024-03-29T11:29:47.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.061+0000 7f1ea6026640 10 monclient: shutdown 2024-03-29T11:29:47.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.061+0000 7f1ea6026640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:47.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.061+0000 7f1ea6026640 1 -- 172.21.15.59:0/3777231755 >> v1:172.21.15.172:6789/0 conn(0x7f1ea00b4820 legacy=0x7f1ea013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.061+0000 7f1ea6026640 1 -- 172.21.15.59:0/3777231755 shutdown_connections 2024-03-29T11:29:47.054 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.061+0000 7f1ea6026640 1 -- 172.21.15.59:0/3777231755 >> 172.21.15.59:0/3777231755 conn(0x7f1ea006d070 msgr2=0x7f1ea0066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.061+0000 7f1ea6026640 1 -- 172.21.15.59:0/3777231755 shutdown_connections 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.061+0000 7f1ea6026640 1 -- 172.21.15.59:0/3777231755 wait complete. 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.062+0000 7f1ea6026640 10 monclient: build_initial_monmap 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.062+0000 7f1ea6026640 10 monclient: monmap: 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:29:47.056 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:29:47.057 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:47.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.063+0000 7f1ea6026640 1 librados: starting msgr at 2024-03-29T11:29:47.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.063+0000 7f1ea6026640 1 librados: starting objecter 2024-03-29T11:29:47.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.063+0000 7f1ea6026640 1 Processor -- start 2024-03-29T11:29:47.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.063+0000 7f1ea6026640 1 -- start start 2024-03-29T11:29:47.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.063+0000 7f1ea6026640 1 librados: setting wanted keys 2024-03-29T11:29:47.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.063+0000 7f1ea6026640 1 librados: calling monclient init 2024-03-29T11:29:47.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.063+0000 7f1ea6026640 10 monclient: init 2024-03-29T11:29:47.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.064+0000 7f1ea6026640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:47.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.064+0000 7f1ea6026640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:47.057 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.064+0000 7f1ea6026640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea00a9ce0 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:47.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.065+0000 7f1ea4823640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1ea00b3cd0 0x7f1ea0134480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:53624/0 (socket says 172.21.15.59:53624) 2024-03-29T11:29:47.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.065+0000 7f1ea4823640 1 -- 172.21.15.59:0/375427273 learned_addr learned my addr 172.21.15.59:0/375427273 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:47.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.065+0000 7f1e95ffb640 10 client.?.objecter ms_handle_connect 0x7f1ea00b3cd0 2024-03-29T11:29:47.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.065+0000 7f1e95ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:47.058 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.065+0000 7f1ea6026640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea00b3cd0 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:47.059 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.067+0000 7f1ea6026640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea00b4820 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:47.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.067+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea013ef80 con 0x7f1ea00a9ce0 2024-03-29T11:29:47.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.067+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea013f1d0 con 0x7f1ea00b3cd0 2024-03-29T11:29:47.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.067+0000 7f1e95ffb640 10 client.?.objecter ms_handle_connect 0x7f1ea00b4820 2024-03-29T11:29:47.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1e95ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:47.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea013b620 con 0x7f1ea00b4820 2024-03-29T11:29:47.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1ea6026640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:47.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214837703 0 0) 0x7f1ea013f1d0 con 0x7f1ea00b3cd0 2024-03-29T11:29:47.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1ea6026640 10 monclient(hunting): authenticate will time out at 1437.182335s 2024-03-29T11:29:47.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1e95ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.060 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1e95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1e95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1e95ffb640 10 monclient(hunting): my global_id is 4594 2024-03-29T11:29:47.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e84004eb0 con 0x7f1ea00b3cd0 2024-03-29T11:29:47.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1e95ffb640 10 client.?.objecter ms_handle_connect 0x7f1ea00a9ce0 2024-03-29T11:29:47.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1e95ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:47.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.068+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279910091 0 0) 0x7f1ea013b620 con 0x7f1ea00b4820 2024-03-29T11:29:47.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.069+0000 7f1e95ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.069+0000 7f1e95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.069+0000 7f1e95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.069+0000 7f1e95ffb640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:29:47.061 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.069+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea013f1d0 con 0x7f1ea00b4820 2024-03-29T11:29:47.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 342305142 0 0) 0x7f1ea013f1d0 con 0x7f1ea00b4820 2024-03-29T11:29:47.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ea013b620 con 0x7f1ea00b4820 2024-03-29T11:29:47.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3535997731 0 0) 0x7f1e84004eb0 con 0x7f1ea00b3cd0 2024-03-29T11:29:47.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ea013f1d0 con 0x7f1ea00b3cd0 2024-03-29T11:29:47.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1e90001790 con 0x7f1ea00b4820 2024-03-29T11:29:47.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.062 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1e98001c60 con 0x7f1ea00b3cd0 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.063 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:47.064 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:47.064 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:47.064 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:47.064 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:47.064 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:47.064 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.064 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:47.064 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:47.064 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:47.064 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:47.064 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1710338951 0 0) 0x7f1ea013b620 con 0x7f1ea00b4820 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 >> v1:172.21.15.172:6789/0 conn(0x7f1ea00b3cd0 legacy=0x7f1ea0134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 >> v1:172.21.15.59:6789/0 conn(0x7f1ea00a9ce0 legacy=0x7f1ea0132e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 monclient: found mon.c 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea013a670 con 0x7f1ea00b4820 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1ea6026640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1ea6026640 10 monclient: _renew_subs 2024-03-29T11:29:47.065 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1ea6026640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea0144d50 con 0x7f1ea00b4820 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e90001c80 con 0x7f1ea00b4820 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1e90003790 con 0x7f1ea00b4820 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e95ffb640 10 monclient: dump: 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:47.066 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:47.067 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:47.067 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:47.067 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1e977fe640 4 set_mon_vals no callback set 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1ea6026640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1ea6026640 10 monclient: _renew_subs 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1ea6026640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea0133710 con 0x7f1ea00b4820 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.070+0000 7f1ea6026640 1 librados: init done 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.072+0000 7f1ea6026640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.072+0000 7f1ea6026640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.072+0000 7f1ea6026640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.072+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea004f730 con 0x7f1ea00b4820 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.074+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f1e90019760 con 0x7f1ea00b4820 2024-03-29T11:29:47.069 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.075+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2848875472 0 0) 0x7f1e90053380 con 0x7f1ea00b4820 2024-03-29T11:29:47.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.075+0000 7f1e95ffb640 10 client.4610.objecter ms_dispatch 0x7f1ea0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:47.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.075+0000 7f1e95ffb640 3 client.4610.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:47.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.075+0000 7f1e95ffb640 3 client.4610.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:47.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.075+0000 7f1e95ffb640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-29T11:29:47.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.076+0000 7f1ea5024640 10 monclient: get_auth_request con 0x7f1e8406e0d0 auth_method 0 2024-03-29T11:29:47.070 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.076+0000 7f1e95ffb640 10 client.4610.objecter ms_handle_connect 0x7f1e8406e0d0 2024-03-29T11:29:47.075 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.084+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e9004f620 con 0x7f1ea00b4820 2024-03-29T11:29:47.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.084+0000 7f1e95ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:47.076 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.084+0000 7f1e95ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:47.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.424+0000 7f1ea6026640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.424+0000 7f1ea6026640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.424+0000 7f1ea6026640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:29:47.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.424+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ea013a800 con 0x7f1ea00b4820 2024-03-29T11:29:47.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.424+0000 7f1e95ffb640 1 -- 172.21.15.59:0/375427273 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2144806415) 0x7f1e9004f620 con 0x7f1ea00b4820 2024-03-29T11:29:47.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.424+0000 7f1e95ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.424+0000 7f1e95ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:47.418 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:29:47.418 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":11,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:29:44.953085+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:44.953085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:47.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.428+0000 7f1ea6026640 10 librados: watch_flush enter 2024-03-29T11:29:47.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.428+0000 7f1ea6026640 10 librados: watch_flush exit 2024-03-29T11:29:47.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.428+0000 7f1ea6026640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-29T11:29:47.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.428+0000 7f1ea6026640 10 client.4610.objecter successfully canceled tick 2024-03-29T11:29:47.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.428+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 >> v1:172.21.15.172:6816/2420742773 conn(0x7f1e8406e0d0 legacy=0x7f1e84070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.428+0000 7f1ea6026640 10 monclient: shutdown 2024-03-29T11:29:47.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.428+0000 7f1ea6026640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:47.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.428+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 >> v1:172.21.15.59:6790/0 conn(0x7f1ea00b4820 legacy=0x7f1ea0134b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.429+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 shutdown_connections 2024-03-29T11:29:47.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.429+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 >> 172.21.15.59:0/375427273 conn(0x7f1ea006d070 msgr2=0x7f1ea00b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:47.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.429+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 shutdown_connections 2024-03-29T11:29:47.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.429+0000 7f1ea6026640 1 -- 172.21.15.59:0/375427273 wait complete. 2024-03-29T11:29:47.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:47.429+0000 7f1ea6026640 1 librados: shutdown 2024-03-29T11:29:47.438 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:54.246 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:54.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.411+0000 7fd02a0a8640 10 monclient: get_monmap_and_config 2024-03-29T11:29:54.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.411+0000 7fd02a0a8640 10 monclient: build_initial_monmap 2024-03-29T11:29:54.405 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.411+0000 7fd02a0a8640 10 monclient: monmap: 2024-03-29T11:29:54.405 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:54.405 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:54.405 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:29:54.412680+0000 2024-03-29T11:29:54.405 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:29:54.412680+0000 2024-03-29T11:29:54.405 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:54.405 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:54.405 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:54.405 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:29:54.406 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:29:54.406 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:54.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.413+0000 7fd02a0a8640 1 Processor -- start 2024-03-29T11:29:54.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.413+0000 7fd02a0a8640 1 -- start start 2024-03-29T11:29:54.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.413+0000 7fd02a0a8640 10 monclient: init 2024-03-29T11:29:54.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.414+0000 7fd02a0a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:54.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.414+0000 7fd02a0a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:54.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.414+0000 7fd02a0a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd0240a9ac0 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:54.406 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.414+0000 7fd02a0a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd02413ee90 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:54.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.414+0000 7fd02a0a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd0240af900 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:54.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.414+0000 7fd02a0a8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0240afd10 con 0x7fd0240af900 2024-03-29T11:29:54.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.414+0000 7fd02a0a8640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0240af6f0 con 0x7fd0240a9ac0 2024-03-29T11:29:54.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.414+0000 7fd02a0a8640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0240aac80 con 0x7fd02413ee90 2024-03-29T11:29:54.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.414+0000 7fd02a0a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:54.407 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.414+0000 7fd02a0a8640 10 monclient(hunting): authenticate will time out at 1444.528035s 2024-03-29T11:29:54.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.420+0000 7fd0237fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd0240a9ac0 0x7fd0240adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:42729/0 (socket says 172.21.15.59:42729) 2024-03-29T11:29:54.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.420+0000 7fd0237fe640 1 -- 172.21.15.59:0/114637846 learned_addr learned my addr 172.21.15.59:0/114637846 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:54.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.420+0000 7fd0227fc640 1 -- 172.21.15.59:0/114637846 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130711753 0 0) 0x7fd0240af6f0 con 0x7fd0240a9ac0 2024-03-29T11:29:54.415 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.420+0000 7fd0227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.420+0000 7fd0227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.420+0000 7fd0227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.420+0000 7fd0227fc640 10 monclient(hunting): my global_id is 4597 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.420+0000 7fd0227fc640 1 -- 172.21.15.59:0/114637846 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd00c003520 con 0x7fd0240a9ac0 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd0227fc640 1 -- 172.21.15.59:0/114637846 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2399232923 0 0) 0x7fd00c003520 con 0x7fd0240a9ac0 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd0227fc640 1 -- 172.21.15.59:0/114637846 >> v1:172.21.15.59:6790/0 conn(0x7fd02413ee90 legacy=0x7fd02413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd0227fc640 1 -- 172.21.15.59:0/114637846 >> v1:172.21.15.59:6789/0 conn(0x7fd0240af900 legacy=0x7fd024149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd0227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd0227fc640 1 monclient: found mon.noname-c 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd0227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd0227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd0227fc640 1 -- 172.21.15.59:0/114637846 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0240aaf60 con 0x7fd0240a9ac0 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd0227fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:54.416 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd0227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd0227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.421+0000 7fd02a0a8640 5 monclient: authenticate success, global_id 4597 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.422+0000 7fd02a0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.422+0000 7fd0227fc640 1 -- 172.21.15.59:0/114637846 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd01c001a10 con 0x7fd0240a9ac0 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.422+0000 7fd0227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.422+0000 7fd0227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.422+0000 7fd0227fc640 10 monclient: dump: 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:54.417 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:54.418 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:54.418 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:54.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.422+0000 7fd02a0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:54.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.422+0000 7fd0227fc640 1 -- 172.21.15.59:0/114637846 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd01c001ca0 con 0x7fd0240a9ac0 2024-03-29T11:29:54.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.422+0000 7fd0227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:54.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.422+0000 7fd02a0a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:54.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.422+0000 7fd02a0a8640 4 set_mon_vals no callback set 2024-03-29T11:29:54.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.423+0000 7fd0227fc640 1 -- 172.21.15.59:0/114637846 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd01c002070 con 0x7fd0240a9ac0 2024-03-29T11:29:54.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.423+0000 7fd0227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.418 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.423+0000 7fd0227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.423+0000 7fd0227fc640 10 monclient: dump: 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.423+0000 7fd02a0a8640 10 monclient: shutdown 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.423+0000 7fd02a0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.423+0000 7fd02a0a8640 1 -- 172.21.15.59:0/114637846 >> v1:172.21.15.172:6789/0 conn(0x7fd0240a9ac0 legacy=0x7fd0240adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.419 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.423+0000 7fd02a0a8640 1 -- 172.21.15.59:0/114637846 shutdown_connections 2024-03-29T11:29:54.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.423+0000 7fd02a0a8640 1 -- 172.21.15.59:0/114637846 >> 172.21.15.59:0/114637846 conn(0x7fd0240b35e0 msgr2=0x7fd0240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:54.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.424+0000 7fd02a0a8640 1 -- 172.21.15.59:0/114637846 shutdown_connections 2024-03-29T11:29:54.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.424+0000 7fd02a0a8640 1 -- 172.21.15.59:0/114637846 wait complete. 2024-03-29T11:29:54.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.425+0000 7fd02a0a8640 10 monclient: build_initial_monmap 2024-03-29T11:29:54.420 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.425+0000 7fd02a0a8640 10 monclient: monmap: 2024-03-29T11:29:54.420 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:29:54.420 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:54.420 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.426+0000 7fd02a0a8640 1 librados: starting msgr at 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.426+0000 7fd02a0a8640 1 librados: starting objecter 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.426+0000 7fd02a0a8640 1 Processor -- start 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.426+0000 7fd02a0a8640 1 -- start start 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.426+0000 7fd02a0a8640 1 librados: setting wanted keys 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.426+0000 7fd02a0a8640 1 librados: calling monclient init 2024-03-29T11:29:54.421 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.426+0000 7fd02a0a8640 10 monclient: init 2024-03-29T11:29:54.424 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.427+0000 7fd02a0a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.427+0000 7fd02a0a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.427+0000 7fd02a0a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd0240a9ac0 addr v1:172.21.15.59:6789/0 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.427+0000 7fd02a0a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd0240af900 addr v1:172.21.15.59:6790/0 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.427+0000 7fd02a0a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd02413ee90 addr v1:172.21.15.172:6789/0 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.427+0000 7fd02a0a8640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0241302f0 con 0x7fd0240a9ac0 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.427+0000 7fd02a0a8640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd024072610 con 0x7fd02413ee90 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.427+0000 7fd02a0a8640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0241c7cd0 con 0x7fd0240af900 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.427+0000 7fd02a0a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.427+0000 7fd02a0a8640 10 monclient(hunting): authenticate will time out at 1444.541087s 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.432+0000 7fd0237fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd0240a9ac0 0x7fd024135e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:56423/0 (socket says 172.21.15.59:56423) 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.432+0000 7fd0237fe640 1 -- 172.21.15.59:0/3784138279 learned_addr learned my addr 172.21.15.59:0/3784138279 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.433+0000 7fd00bfff640 10 client.?.objecter ms_handle_connect 0x7fd0240a9ac0 2024-03-29T11:29:54.425 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.433+0000 7fd00bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:54.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 10 client.?.objecter ms_handle_connect 0x7fd02413ee90 2024-03-29T11:29:54.430 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:54.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106854878 0 0) 0x7fd0241302f0 con 0x7fd0240a9ac0 2024-03-29T11:29:54.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.433 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd010004eb0 con 0x7fd0240a9ac0 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798933469 0 0) 0x7fd024072610 con 0x7fd02413ee90 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 10 monclient(hunting): my global_id is 4600 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.435+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0241302f0 con 0x7fd02413ee90 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.436+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1026728490 0 0) 0x7fd010004eb0 con 0x7fd0240a9ac0 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.436+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd024072610 con 0x7fd0240a9ac0 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.436+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd01c001990 con 0x7fd0240a9ac0 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.436+0000 7fd00bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.436+0000 7fd00bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:54.434 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.436+0000 7fd00bfff640 10 monclient(hunting): dump: 2024-03-29T11:29:54.435 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:54.435 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:54.435 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:54.435 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:54.435 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.435 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:54.435 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:54.435 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:54.436 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:54.436 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:54.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.436+0000 7fd00bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:54.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.436+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1622094897 0 0) 0x7fd024072610 con 0x7fd0240a9ac0 2024-03-29T11:29:54.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.436+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 >> v1:172.21.15.59:6790/0 conn(0x7fd0240af900 legacy=0x7fd0241c75c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:54.436 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 >> v1:172.21.15.172:6789/0 conn(0x7fd02413ee90 legacy=0x7fd0241ffc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd00bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd00bfff640 1 monclient: found mon.a 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd00bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd00bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0241c9410 con 0x7fd0240a9ac0 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd00bfff640 10 monclient: _finish_auth 0 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd00bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd00bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd02a0a8640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd02a0a8640 10 monclient: _renew_subs 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd02a0a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd02a0a8640 1 -- 172.21.15.59:0/3784138279 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0241c9750 con 0x7fd0240a9ac0 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd02a0a8640 10 client.4590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd02a0a8640 10 monclient: _renew_subs 2024-03-29T11:29:54.437 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd02a0a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:54.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd02a0a8640 1 -- 172.21.15.59:0/3784138279 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0241cd2c0 con 0x7fd0240a9ac0 2024-03-29T11:29:54.438 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.437+0000 7fd02a0a8640 1 librados: init done 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.439+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd01c004f80 con 0x7fd0240a9ac0 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.439+0000 7fd00bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.439+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7fd01c005290 con 0x7fd0240a9ac0 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.439+0000 7fd00bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.439+0000 7fd00bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.439+0000 7fd00bfff640 10 monclient: dump: 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:29:54.440 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.439+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7fd01c01b260 con 0x7fd0240a9ac0 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.440+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2848875472 0 0) 0x7fd01c0850c0 con 0x7fd0240a9ac0 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.440+0000 7fd00bfff640 10 client.4590.objecter ms_dispatch 0x7fd024000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.440+0000 7fd00bfff640 3 client.4590.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.440+0000 7fd00bfff640 3 client.4590.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.440+0000 7fd00bfff640 20 client.4590.objecter dump_active .. 0 homeless 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.440+0000 7fd021ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.443+0000 7fd02a0a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.443+0000 7fd02a0a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.443+0000 7fd02a0a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.443+0000 7fd02a0a8640 1 -- 172.21.15.59:0/3784138279 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd004005080 con 0x7fd0240a9ac0 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.443+0000 7fd022ffd640 10 monclient: get_auth_request con 0x7fd01006db70 auth_method 0 2024-03-29T11:29:54.441 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.443+0000 7fd00bfff640 10 client.4590.objecter ms_handle_connect 0x7fd01006db70 2024-03-29T11:29:54.443 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.451+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd01c050a20 con 0x7fd0240a9ac0 2024-03-29T11:29:54.443 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.451+0000 7fd00bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.443 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.451+0000 7fd00bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:54.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.791+0000 7fd02a0a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.791+0000 7fd02a0a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.791+0000 7fd02a0a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:29:54.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.791+0000 7fd02a0a8640 1 -- 172.21.15.59:0/3784138279 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd004005330 con 0x7fd0240a9ac0 2024-03-29T11:29:54.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.794+0000 7fd00bfff640 1 -- 172.21.15.59:0/3784138279 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2144806415) 0x7fd01c052950 con 0x7fd0240a9ac0 2024-03-29T11:29:54.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.794+0000 7fd00bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.794+0000 7fd00bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:54.787 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:29:54.787 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":11,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:29:44.953085+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:44.953085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:54.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.798+0000 7fd02a0a8640 10 librados: watch_flush enter 2024-03-29T11:29:54.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.798+0000 7fd02a0a8640 10 librados: watch_flush exit 2024-03-29T11:29:54.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.798+0000 7fd02a0a8640 20 client.4590.objecter shutdown clearing up homeless session... 2024-03-29T11:29:54.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.798+0000 7fd02a0a8640 10 client.4590.objecter successfully canceled tick 2024-03-29T11:29:54.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.798+0000 7fd02a0a8640 1 -- 172.21.15.59:0/3784138279 >> v1:172.21.15.172:6816/2420742773 conn(0x7fd01006db70 legacy=0x7fd010070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.798+0000 7fd02a0a8640 10 monclient: shutdown 2024-03-29T11:29:54.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.798+0000 7fd02a0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:54.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.798+0000 7fd02a0a8640 1 -- 172.21.15.59:0/3784138279 >> v1:172.21.15.59:6789/0 conn(0x7fd0240a9ac0 legacy=0x7fd024135e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.793 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.801+0000 7fd02a0a8640 1 -- 172.21.15.59:0/3784138279 shutdown_connections 2024-03-29T11:29:54.793 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.801+0000 7fd02a0a8640 1 -- 172.21.15.59:0/3784138279 >> 172.21.15.59:0/3784138279 conn(0x7fd0240b35e0 msgr2=0x7fd0240acf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:54.796 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.804+0000 7fd02a0a8640 1 -- 172.21.15.59:0/3784138279 shutdown_connections 2024-03-29T11:29:54.796 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.804+0000 7fd02a0a8640 1 -- 172.21.15.59:0/3784138279 wait complete. 2024-03-29T11:29:54.796 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:29:54.804+0000 7fd02a0a8640 1 librados: shutdown 2024-03-29T11:29:54.814 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:01.620 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:01.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.782+0000 7ff8a676b640 10 monclient: get_monmap_and_config 2024-03-29T11:30:01.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.782+0000 7ff8a676b640 10 monclient: build_initial_monmap 2024-03-29T11:30:01.778 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.782+0000 7ff8a676b640 10 monclient: monmap: 2024-03-29T11:30:01.778 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:01.778 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:01.778 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:30:01.784228+0000 2024-03-29T11:30:01.778 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:30:01.784228+0000 2024-03-29T11:30:01.778 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:01.779 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:01.779 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:01.779 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:30:01.779 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:30:01.779 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:01.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.784+0000 7ff8a676b640 1 Processor -- start 2024-03-29T11:30:01.779 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.784+0000 7ff8a676b640 1 -- start start 2024-03-29T11:30:01.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.784+0000 7ff8a676b640 10 monclient: init 2024-03-29T11:30:01.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.785+0000 7ff8a676b640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:01.780 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.785+0000 7ff8a676b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:01.783 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.789+0000 7ff8a676b640 10 monclient(hunting): picked mon.noname-c con 0x7ff8a00ae250 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:01.784 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.789+0000 7ff8a676b640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a013f970 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:01.784 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.789+0000 7ff8a676b640 10 monclient(hunting): picked mon.noname-b con 0x7ff8a0142800 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:01.784 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.789+0000 7ff8a676b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00afe00 con 0x7ff8a013f970 2024-03-29T11:30:01.784 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.789+0000 7ff8a676b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a0142c10 con 0x7ff8a00ae250 2024-03-29T11:30:01.784 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.789+0000 7ff8a676b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00afb20 con 0x7ff8a0142800 2024-03-29T11:30:01.785 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.789+0000 7ff8a676b640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:01.785 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.789+0000 7ff8a676b640 10 monclient(hunting): authenticate will time out at 1451.903811s 2024-03-29T11:30:01.785 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.790+0000 7ff89ffff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff8a00ae250 0x7ff8a013f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:36097/0 (socket says 172.21.15.59:36097) 2024-03-29T11:30:01.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.790+0000 7ff89ffff640 1 -- 172.21.15.59:0/2012626518 learned_addr learned my addr 172.21.15.59:0/2012626518 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:01.786 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.792+0000 7ff89effd640 1 -- 172.21.15.59:0/2012626518 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926034448 0 0) 0x7ff8a0142c10 con 0x7ff8a00ae250 2024-03-29T11:30:01.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.792+0000 7ff89effd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.792+0000 7ff89effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.792+0000 7ff89effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.792+0000 7ff89effd640 10 monclient(hunting): my global_id is 4603 2024-03-29T11:30:01.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.792+0000 7ff89effd640 1 -- 172.21.15.59:0/2012626518 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff888003520 con 0x7ff8a00ae250 2024-03-29T11:30:01.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 1 -- 172.21.15.59:0/2012626518 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866729211 0 0) 0x7ff8a00afe00 con 0x7ff8a013f970 2024-03-29T11:30:01.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:30:01.787 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 1 -- 172.21.15.59:0/2012626518 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a0142c10 con 0x7ff8a013f970 2024-03-29T11:30:01.789 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 1 -- 172.21.15.59:0/2012626518 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3223562819 0 0) 0x7ff8a0142c10 con 0x7ff8a013f970 2024-03-29T11:30:01.789 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 1 -- 172.21.15.59:0/2012626518 >> v1:172.21.15.59:6790/0 conn(0x7ff8a0142800 legacy=0x7ff8a014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:01.789 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 1 -- 172.21.15.59:0/2012626518 >> v1:172.21.15.172:6789/0 conn(0x7ff8a00ae250 legacy=0x7ff8a013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.789 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:01.789 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 1 monclient: found mon.noname-a 2024-03-29T11:30:01.789 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:01.789 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:30:01.789 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 1 -- 172.21.15.59:0/2012626518 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a00aab80 con 0x7ff8a013f970 2024-03-29T11:30:01.789 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 10 monclient: _finish_auth 0 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 1 -- 172.21.15.59:0/2012626518 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff890001670 con 0x7ff8a013f970 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.793+0000 7ff89effd640 10 monclient: dump: 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:01.790 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:01.791 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:01.791 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:01.791 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.796+0000 7ff8a676b640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:30:01.791 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.796+0000 7ff8a676b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:01.791 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.796+0000 7ff89effd640 1 -- 172.21.15.59:0/2012626518 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8900017d0 con 0x7ff8a013f970 2024-03-29T11:30:01.791 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.796+0000 7ff89effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.796+0000 7ff89effd640 1 -- 172.21.15.59:0/2012626518 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff890001c20 con 0x7ff8a013f970 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.796+0000 7ff89effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.796+0000 7ff89effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.796+0000 7ff89effd640 10 monclient: dump: 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:01.792 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.796+0000 7ff8a676b640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:01.793 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.797+0000 7ff8a676b640 4 set_mon_vals no callback set 2024-03-29T11:30:01.793 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.797+0000 7ff8a676b640 10 monclient: shutdown 2024-03-29T11:30:01.793 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.797+0000 7ff8a676b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:01.793 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.797+0000 7ff8a676b640 1 -- 172.21.15.59:0/2012626518 >> v1:172.21.15.59:6789/0 conn(0x7ff8a013f970 legacy=0x7ff8a01420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.793 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.800+0000 7ff8a676b640 1 -- 172.21.15.59:0/2012626518 shutdown_connections 2024-03-29T11:30:01.793 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.800+0000 7ff8a676b640 1 -- 172.21.15.59:0/2012626518 >> 172.21.15.59:0/2012626518 conn(0x7ff8a00b35e0 msgr2=0x7ff8a00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:01.794 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.801+0000 7ff8a676b640 1 -- 172.21.15.59:0/2012626518 shutdown_connections 2024-03-29T11:30:01.795 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.801+0000 7ff8a676b640 1 -- 172.21.15.59:0/2012626518 wait complete. 2024-03-29T11:30:01.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.802+0000 7ff8a676b640 10 monclient: build_initial_monmap 2024-03-29T11:30:01.797 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.802+0000 7ff8a676b640 10 monclient: monmap: 2024-03-29T11:30:01.797 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:01.797 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:01.797 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:30:01.797 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:30:01.797 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:01.797 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:01.798 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:01.798 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:30:01.798 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:30:01.798 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:01.798 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.802+0000 7ff8a676b640 1 librados: starting msgr at 2024-03-29T11:30:01.799 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.802+0000 7ff8a676b640 1 librados: starting objecter 2024-03-29T11:30:01.799 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.802+0000 7ff8a676b640 1 Processor -- start 2024-03-29T11:30:01.799 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.802+0000 7ff8a676b640 1 -- start start 2024-03-29T11:30:01.799 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.802+0000 7ff8a676b640 1 librados: setting wanted keys 2024-03-29T11:30:01.799 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.802+0000 7ff8a676b640 1 librados: calling monclient init 2024-03-29T11:30:01.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.802+0000 7ff8a676b640 10 monclient: init 2024-03-29T11:30:01.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.803+0000 7ff8a676b640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:01.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.803+0000 7ff8a676b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:01.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.803+0000 7ff8a676b640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a00ae250 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:01.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.803+0000 7ff8a676b640 10 monclient(hunting): picked mon.noname-b con 0x7ff8a013f970 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:01.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.803+0000 7ff8a676b640 10 monclient(hunting): picked mon.noname-c con 0x7ff8a0142800 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:01.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.803+0000 7ff8a676b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a0146140 con 0x7ff8a00ae250 2024-03-29T11:30:01.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.803+0000 7ff8a676b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a0143d50 con 0x7ff8a013f970 2024-03-29T11:30:01.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.803+0000 7ff8a676b640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a01e3d30 con 0x7ff8a0142800 2024-03-29T11:30:01.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.803+0000 7ff8a676b640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:01.800 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.803+0000 7ff8a676b640 10 monclient(hunting): authenticate will time out at 1451.918402s 2024-03-29T11:30:01.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.810+0000 7ff89ffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff8a00ae250 0x7ff8a01447a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58865/0 (socket says 172.21.15.59:58865) 2024-03-29T11:30:01.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.810+0000 7ff89ffff640 1 -- 172.21.15.59:0/649443267 learned_addr learned my addr 172.21.15.59:0/649443267 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:01.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.810+0000 7ff89cff9640 10 client.?.objecter ms_handle_connect 0x7ff8a00ae250 2024-03-29T11:30:01.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.810+0000 7ff89cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:01.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.812+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776724174 0 0) 0x7ff8a0146140 con 0x7ff8a00ae250 2024-03-29T11:30:01.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.812+0000 7ff89cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.812+0000 7ff89cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.812+0000 7ff89cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.812+0000 7ff89cff9640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.812+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff88c004eb0 con 0x7ff8a00ae250 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.812+0000 7ff89cff9640 10 client.?.objecter ms_handle_connect 0x7ff8a0142800 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.812+0000 7ff89cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 client.?.objecter ms_handle_connect 0x7ff8a013f970 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453801295 0 0) 0x7ff8a0143d50 con 0x7ff8a013f970 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): my global_id is 4606 2024-03-29T11:30:01.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a0146140 con 0x7ff8a013f970 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363192312 0 0) 0x7ff8a01e3d30 con 0x7ff8a0142800 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a0143d50 con 0x7ff8a0142800 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3750417892 0 0) 0x7ff8a0143d50 con 0x7ff8a0142800 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8a01e3d30 con 0x7ff8a0142800 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff894001610 con 0x7ff8a0142800 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): dump: 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:01.809 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2640350808 0 0) 0x7ff8a01e3d30 con 0x7ff8a0142800 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 >> v1:172.21.15.172:6789/0 conn(0x7ff8a013f970 legacy=0x7ff8a01e4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 >> v1:172.21.15.59:6789/0 conn(0x7ff8a00ae250 legacy=0x7ff8a01447a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 monclient: found mon.c 2024-03-29T11:30:01.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:01.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:01.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a0144eb0 con 0x7ff8a0142800 2024-03-29T11:30:01.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient: _finish_auth 0 2024-03-29T11:30:01.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:30:01.811 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.813+0000 7ff89cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:01.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.814+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff894001b20 con 0x7ff8a0142800 2024-03-29T11:30:01.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.814+0000 7ff89cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:01.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.814+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7ff894003790 con 0x7ff8a0142800 2024-03-29T11:30:01.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.814+0000 7ff89cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.814+0000 7ff89cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:01.813 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.814+0000 7ff89cff9640 10 monclient: dump: 2024-03-29T11:30:01.813 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:01.813 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.814+0000 7ff89e7fc640 4 set_mon_vals no callback set 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.815+0000 7ff8a676b640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.815+0000 7ff8a676b640 10 monclient: _renew_subs 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.815+0000 7ff8a676b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.815+0000 7ff8a676b640 1 -- 172.21.15.59:0/649443267 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a0145120 con 0x7ff8a0142800 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.815+0000 7ff8a676b640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:01.814 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.815+0000 7ff8a676b640 10 monclient: _renew_subs 2024-03-29T11:30:01.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.815+0000 7ff8a676b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:01.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.815+0000 7ff8a676b640 1 -- 172.21.15.59:0/649443267 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a013ac80 con 0x7ff8a0142800 2024-03-29T11:30:01.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.815+0000 7ff8a676b640 1 librados: init done 2024-03-29T11:30:01.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.817+0000 7ff8a676b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:01.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.817+0000 7ff8a676b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:01.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.817+0000 7ff8a676b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:01.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.817+0000 7ff8a676b640 1 -- 172.21.15.59:0/649443267 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a0075f10 con 0x7ff8a0142800 2024-03-29T11:30:01.815 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.820+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7ff894019760 con 0x7ff8a0142800 2024-03-29T11:30:01.816 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.821+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2848875472 0 0) 0x7ff8940532b0 con 0x7ff8a0142800 2024-03-29T11:30:01.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.821+0000 7ff89cff9640 10 client.4613.objecter ms_dispatch 0x7ff8a0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:01.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.821+0000 7ff89cff9640 3 client.4613.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:01.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.821+0000 7ff89cff9640 3 client.4613.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:01.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.821+0000 7ff89cff9640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:30:01.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.821+0000 7ff89ffff640 10 monclient: get_auth_request con 0x7ff88c06db50 auth_method 0 2024-03-29T11:30:01.817 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.822+0000 7ff89cff9640 10 client.4613.objecter ms_handle_connect 0x7ff88c06db50 2024-03-29T11:30:01.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.826+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff89404f550 con 0x7ff8a0142800 2024-03-29T11:30:01.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.826+0000 7ff89cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:01.819 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:01.826+0000 7ff89cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:02.196 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.203+0000 7ff8a676b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:02.196 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.203+0000 7ff8a676b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:02.196 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.203+0000 7ff8a676b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:02.197 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.203+0000 7ff8a676b640 1 -- 172.21.15.59:0/649443267 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8a01452b0 con 0x7ff8a0142800 2024-03-29T11:30:02.198 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.205+0000 7ff89cff9640 1 -- 172.21.15.59:0/649443267 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2144806415) 0x7ff89404f6f0 con 0x7ff8a0142800 2024-03-29T11:30:02.198 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.205+0000 7ff89cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:02.198 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.205+0000 7ff89cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:02.198 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:30:02.198 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":11,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:29:44.953085+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:44.953085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:02.202 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.209+0000 7ff8a676b640 10 librados: watch_flush enter 2024-03-29T11:30:02.202 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.209+0000 7ff8a676b640 10 librados: watch_flush exit 2024-03-29T11:30:02.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.209+0000 7ff8a676b640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:30:02.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.209+0000 7ff8a676b640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:30:02.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.209+0000 7ff8a676b640 1 -- 172.21.15.59:0/649443267 >> v1:172.21.15.172:6816/2420742773 conn(0x7ff88c06db50 legacy=0x7ff88c070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.209+0000 7ff8a676b640 10 monclient: shutdown 2024-03-29T11:30:02.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.209+0000 7ff8a676b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:02.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.209+0000 7ff8a676b640 1 -- 172.21.15.59:0/649443267 >> v1:172.21.15.59:6790/0 conn(0x7ff8a0142800 legacy=0x7ff8a0071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.210+0000 7ff8a676b640 1 -- 172.21.15.59:0/649443267 shutdown_connections 2024-03-29T11:30:02.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.210+0000 7ff8a676b640 1 -- 172.21.15.59:0/649443267 >> 172.21.15.59:0/649443267 conn(0x7ff8a00b35e0 msgr2=0x7ff8a00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:02.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.210+0000 7ff8a676b640 1 -- 172.21.15.59:0/649443267 shutdown_connections 2024-03-29T11:30:02.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.210+0000 7ff8a676b640 1 -- 172.21.15.59:0/649443267 wait complete. 2024-03-29T11:30:02.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:02.210+0000 7ff8a676b640 1 librados: shutdown 2024-03-29T11:30:02.218 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:09.026 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:09.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.182+0000 7f5fd40d9640 10 monclient: get_monmap_and_config 2024-03-29T11:30:09.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.182+0000 7f5fd40d9640 10 monclient: build_initial_monmap 2024-03-29T11:30:09.176 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.182+0000 7f5fd40d9640 10 monclient: monmap: 2024-03-29T11:30:09.176 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:09.176 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:09.176 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:30:09.184605+0000 2024-03-29T11:30:09.176 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:30:09.184605+0000 2024-03-29T11:30:09.177 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:09.177 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:09.177 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:09.177 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:30:09.177 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:30:09.177 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:09.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.184+0000 7f5fd40d9640 1 Processor -- start 2024-03-29T11:30:09.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.185+0000 7f5fd40d9640 1 -- start start 2024-03-29T11:30:09.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.185+0000 7f5fd40d9640 10 monclient: init 2024-03-29T11:30:09.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.186+0000 7f5fd40d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:09.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.186+0000 7f5fd40d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:09.179 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.186+0000 7f5fd40d9640 10 monclient(hunting): picked mon.noname-a con 0x7f5fcc0b0860 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:09.182 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.189+0000 7f5fd40d9640 10 monclient(hunting): picked mon.noname-c con 0x7f5fcc06d9d0 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:09.182 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.189+0000 7f5fd40d9640 10 monclient(hunting): picked mon.noname-b con 0x7f5fcc0afa60 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:09.182 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.189+0000 7f5fd40d9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc0b0c40 con 0x7f5fcc0b0860 2024-03-29T11:30:09.182 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.189+0000 7f5fd40d9640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc0aed10 con 0x7f5fcc06d9d0 2024-03-29T11:30:09.182 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.189+0000 7f5fd40d9640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc0ab1b0 con 0x7f5fcc0afa60 2024-03-29T11:30:09.182 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.189+0000 7f5fd40d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:09.182 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.189+0000 7f5fd40d9640 10 monclient(hunting): authenticate will time out at 1459.304205s 2024-03-29T11:30:09.185 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.192+0000 7f5fd1e75640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f5fcc0b0860 0x7f5fcc13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49443/0 (socket says 172.21.15.59:49443) 2024-03-29T11:30:09.185 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.192+0000 7f5fd1e75640 1 -- 172.21.15.59:0/1958549964 learned_addr learned my addr 172.21.15.59:0/1958549964 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:09.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 1 -- 172.21.15.59:0/1958549964 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627766708 0 0) 0x7f5fcc0aed10 con 0x7f5fcc06d9d0 2024-03-29T11:30:09.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 10 monclient(hunting): _negotiate 2024-03-29T11:30:09.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:09.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:09.191 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 10 monclient(hunting): my global_id is 4609 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 1 -- 172.21.15.59:0/1958549964 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb8003520 con 0x7f5fcc06d9d0 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 1 -- 172.21.15.59:0/1958549964 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490724411 0 0) 0x7f5fcc0ab1b0 con 0x7f5fcc0afa60 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 10 monclient(hunting): _negotiate 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.195+0000 7f5fd0e73640 1 -- 172.21.15.59:0/1958549964 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fcc0aed10 con 0x7f5fcc0afa60 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 1 -- 172.21.15.59:0/1958549964 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3054464910 0 0) 0x7f5fb8003520 con 0x7f5fcc06d9d0 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 1 -- 172.21.15.59:0/1958549964 >> v1:172.21.15.59:6790/0 conn(0x7f5fcc0afa60 legacy=0x7f5fcc14a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 1 -- 172.21.15.59:0/1958549964 >> v1:172.21.15.59:6789/0 conn(0x7f5fcc0b0860 legacy=0x7f5fcc13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 10 monclient: _finish_hunting 0 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 1 monclient: found mon.noname-c 2024-03-29T11:30:09.192 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:09.193 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.172:6789/0 2024-03-29T11:30:09.193 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 1 -- 172.21.15.59:0/1958549964 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc0ab490 con 0x7f5fcc06d9d0 2024-03-29T11:30:09.193 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 10 monclient: _finish_auth 0 2024-03-29T11:30:09.193 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 10 monclient: _check_auth_tickets 2024-03-29T11:30:09.193 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:09.193 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 1 -- 172.21.15.59:0/1958549964 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5fbc001690 con 0x7f5fcc06d9d0 2024-03-29T11:30:09.193 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:09.193 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:09.193 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd0e73640 10 monclient: dump: 2024-03-29T11:30:09.193 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:09.195 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:09.195 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:09.195 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:09.195 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:09.195 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:09.196 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:09.196 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:09.196 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:09.196 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:09.196 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd40d9640 5 monclient: authenticate success, global_id 4609 2024-03-29T11:30:09.196 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.196+0000 7f5fd40d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:09.198 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.201+0000 7f5fd0e73640 1 -- 172.21.15.59:0/1958549964 <== mon.1 v1:172.21.15.172:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fbc0017f0 con 0x7f5fcc06d9d0 2024-03-29T11:30:09.198 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.201+0000 7f5fd0e73640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:09.198 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.201+0000 7f5fd0e73640 1 -- 172.21.15.59:0/1958549964 <== mon.1 v1:172.21.15.172:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5fbc001c00 con 0x7f5fcc06d9d0 2024-03-29T11:30:09.198 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.201+0000 7f5fd0e73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.201+0000 7f5fd0e73640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.201+0000 7f5fd0e73640 10 monclient: dump: 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:09.199 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.201+0000 7f5fd40d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:09.200 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.201+0000 7f5fd40d9640 4 set_mon_vals no callback set 2024-03-29T11:30:09.200 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.202+0000 7f5fd40d9640 10 monclient: shutdown 2024-03-29T11:30:09.200 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.202+0000 7f5fd40d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:09.200 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.202+0000 7f5fd40d9640 1 -- 172.21.15.59:0/1958549964 >> v1:172.21.15.172:6789/0 conn(0x7f5fcc06d9d0 legacy=0x7f5fcc146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.200 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.203+0000 7f5fd40d9640 1 -- 172.21.15.59:0/1958549964 shutdown_connections 2024-03-29T11:30:09.200 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.203+0000 7f5fd40d9640 1 -- 172.21.15.59:0/1958549964 >> 172.21.15.59:0/1958549964 conn(0x7f5fcc06b660 msgr2=0x7f5fcc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:09.200 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.204+0000 7f5fd40d9640 1 -- 172.21.15.59:0/1958549964 shutdown_connections 2024-03-29T11:30:09.201 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.205+0000 7f5fd40d9640 1 -- 172.21.15.59:0/1958549964 wait complete. 2024-03-29T11:30:09.201 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.206+0000 7f5fd40d9640 10 monclient: build_initial_monmap 2024-03-29T11:30:09.201 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.206+0000 7f5fd40d9640 10 monclient: monmap: 2024-03-29T11:30:09.201 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.206+0000 7f5fd40d9640 1 librados: starting msgr at 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.206+0000 7f5fd40d9640 1 librados: starting objecter 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.207+0000 7f5fd40d9640 1 Processor -- start 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.207+0000 7f5fd40d9640 1 -- start start 2024-03-29T11:30:09.202 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.207+0000 7f5fd40d9640 1 librados: setting wanted keys 2024-03-29T11:30:09.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.207+0000 7f5fd40d9640 1 librados: calling monclient init 2024-03-29T11:30:09.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.207+0000 7f5fd40d9640 10 monclient: init 2024-03-29T11:30:09.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.208+0000 7f5fd40d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:09.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.208+0000 7f5fd40d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:09.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.208+0000 7f5fd40d9640 10 monclient(hunting): picked mon.noname-b con 0x7f5fcc06d9d0 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:09.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.208+0000 7f5fd40d9640 10 monclient(hunting): picked mon.noname-a con 0x7f5fcc0afa60 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:09.203 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.208+0000 7f5fd40d9640 10 monclient(hunting): picked mon.noname-c con 0x7f5fcc0b0860 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:09.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.208+0000 7f5fd40d9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc0b2f00 con 0x7f5fcc0afa60 2024-03-29T11:30:09.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.208+0000 7f5fd40d9640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc0b31e0 con 0x7f5fcc06d9d0 2024-03-29T11:30:09.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.208+0000 7f5fd40d9640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc14b5c0 con 0x7f5fcc0b0860 2024-03-29T11:30:09.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.208+0000 7f5fd40d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:09.204 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.208+0000 7f5fd40d9640 10 monclient(hunting): authenticate will time out at 1459.323297s 2024-03-29T11:30:09.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.215+0000 7f5fd2676640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f5fcc0b0860 0x7f5fcc1e6340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:46359/0 (socket says 172.21.15.59:46359) 2024-03-29T11:30:09.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.215+0000 7f5fd2676640 1 -- 172.21.15.59:0/2205975757 learned_addr learned my addr 172.21.15.59:0/2205975757 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:09.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.215+0000 7f5fc27fc640 10 client.?.objecter ms_handle_connect 0x7f5fcc0b0860 2024-03-29T11:30:09.208 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.215+0000 7f5fc27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:09.213 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 10 client.?.objecter ms_handle_connect 0x7f5fcc0afa60 2024-03-29T11:30:09.213 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450078907 0 0) 0x7f5fcc14b5c0 con 0x7f5fcc0b0860 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb0004eb0 con 0x7f5fcc0b0860 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676567290 0 0) 0x7f5fcc0b2f00 con 0x7f5fcc0afa60 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.216+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb0006230 con 0x7f5fcc0afa60 2024-03-29T11:30:09.214 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.217+0000 7f5fc27fc640 10 client.?.objecter ms_handle_connect 0x7f5fcc06d9d0 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.217+0000 7f5fc27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.217+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 764439409 0 0) 0x7f5fb0006230 con 0x7f5fcc0afa60 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.217+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fb00066a0 con 0x7f5fcc0afa60 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.217+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5fbc001b00 con 0x7f5fcc0afa60 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.217+0000 7f5fc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.217+0000 7f5fc27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.217+0000 7f5fc27fc640 10 monclient(hunting): dump: 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:09.215 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.217+0000 7f5fc27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760545963 0 0) 0x7f5fcc0b31e0 con 0x7f5fcc06d9d0 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 10 monclient(hunting): my global_id is 4612 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb0006bf0 con 0x7f5fcc06d9d0 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.0 v1:172.21.15.59:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1711040012 0 0) 0x7f5fb00066a0 con 0x7f5fcc0afa60 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 >> v1:172.21.15.59:6790/0 conn(0x7f5fcc0b0860 legacy=0x7f5fcc1e6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 >> v1:172.21.15.172:6789/0 conn(0x7f5fcc06d9d0 legacy=0x7f5fcc0b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:09.216 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 1 monclient: found mon.a 2024-03-29T11:30:09.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:09.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:30:09.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc0b3d10 con 0x7f5fcc0afa60 2024-03-29T11:30:09.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:09.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:09.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:09.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fd40d9640 5 monclient: authenticate success, global_id 4602 2024-03-29T11:30:09.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fd40d9640 10 monclient: _renew_subs 2024-03-29T11:30:09.217 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fd40d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:30:09.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fd40d9640 1 -- 172.21.15.59:0/2205975757 --> v1:172.21.15.59:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fcc0b4270 con 0x7f5fcc0afa60 2024-03-29T11:30:09.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fd40d9640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:09.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fd40d9640 10 monclient: _renew_subs 2024-03-29T11:30:09.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fd40d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:30:09.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.218+0000 7f5fd40d9640 1 -- 172.21.15.59:0/2205975757 --> v1:172.21.15.59:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fcc1382f0 con 0x7f5fcc0afa60 2024-03-29T11:30:09.218 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.219+0000 7f5fd40d9640 1 librados: init done 2024-03-29T11:30:09.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.221+0000 7f5fd40d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:09.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.221+0000 7f5fd40d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:09.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.221+0000 7f5fd40d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:30:09.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.221+0000 7f5fd40d9640 1 -- 172.21.15.59:0/2205975757 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fcc075f10 con 0x7f5fcc0afa60 2024-03-29T11:30:09.223 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.225+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.0 v1:172.21.15.59:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fbc001240 con 0x7f5fcc0afa60 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.225+0000 7f5fc27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.225+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.0 v1:172.21.15.59:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f5fbc001b00 con 0x7f5fcc0afa60 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.225+0000 7f5fc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.225+0000 7f5fc27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.225+0000 7f5fc27fc640 10 monclient: dump: 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:09.224 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:09.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.225+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.0 v1:172.21.15.59:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f5fbc019710 con 0x7f5fcc0afa60 2024-03-29T11:30:09.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.226+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.0 v1:172.21.15.59:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2848875472 0 0) 0x7f5fbc0835d0 con 0x7f5fcc0afa60 2024-03-29T11:30:09.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.226+0000 7f5fc27fc640 10 client.4602.objecter ms_dispatch 0x7f5fcc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:09.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.226+0000 7f5fc27fc640 3 client.4602.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:09.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.226+0000 7f5fc27fc640 3 client.4602.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:09.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.226+0000 7f5fc27fc640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-29T11:30:09.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.228+0000 7f5fc3fff640 4 set_mon_vals no callback set 2024-03-29T11:30:09.225 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.228+0000 7f5fd1e75640 10 monclient: get_auth_request con 0x7f5fb006e670 auth_method 0 2024-03-29T11:30:09.226 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.230+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.0 v1:172.21.15.59:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5fbc04ef30 con 0x7f5fcc0afa60 2024-03-29T11:30:09.226 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.230+0000 7f5fc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:09.226 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.230+0000 7f5fc27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:09.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.238+0000 7f5fc27fc640 10 client.4602.objecter ms_handle_connect 0x7f5fb006e670 2024-03-29T11:30:09.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.555+0000 7f5fd40d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:09.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.555+0000 7f5fd40d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:09.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.555+0000 7f5fd40d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.59:6789/0 2024-03-29T11:30:09.548 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.555+0000 7f5fd40d9640 1 -- 172.21.15.59:0/2205975757 --> v1:172.21.15.59:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fcc0b3fd0 con 0x7f5fcc0afa60 2024-03-29T11:30:09.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.559+0000 7f5fc27fc640 1 -- 172.21.15.59:0/2205975757 <== mon.0 v1:172.21.15.59:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2144806415) 0x7f5fbc050e60 con 0x7f5fcc0afa60 2024-03-29T11:30:09.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.559+0000 7f5fc27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:09.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.559+0000 7f5fc27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:09.555 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:30:09.556 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":11,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:29:44.953085+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:44.953085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:09.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.564+0000 7f5fd40d9640 10 librados: watch_flush enter 2024-03-29T11:30:09.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.564+0000 7f5fd40d9640 10 librados: watch_flush exit 2024-03-29T11:30:09.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.564+0000 7f5fd40d9640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-29T11:30:09.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.564+0000 7f5fd40d9640 10 client.4602.objecter successfully canceled tick 2024-03-29T11:30:09.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.564+0000 7f5fd40d9640 1 -- 172.21.15.59:0/2205975757 >> v1:172.21.15.172:6816/2420742773 conn(0x7f5fb006e670 legacy=0x7f5fb0070b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.564+0000 7f5fd40d9640 10 monclient: shutdown 2024-03-29T11:30:09.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.564+0000 7f5fd40d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:09.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.565+0000 7f5fd40d9640 1 -- 172.21.15.59:0/2205975757 >> v1:172.21.15.59:6789/0 conn(0x7f5fcc0afa60 legacy=0x7f5fcc0b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.565+0000 7f5fd40d9640 1 -- 172.21.15.59:0/2205975757 shutdown_connections 2024-03-29T11:30:09.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.565+0000 7f5fd40d9640 1 -- 172.21.15.59:0/2205975757 >> 172.21.15.59:0/2205975757 conn(0x7f5fcc06b660 msgr2=0x7f5fcc0af0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:09.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.565+0000 7f5fd40d9640 1 -- 172.21.15.59:0/2205975757 shutdown_connections 2024-03-29T11:30:09.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.565+0000 7f5fd40d9640 1 -- 172.21.15.59:0/2205975757 wait complete. 2024-03-29T11:30:09.559 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:09.565+0000 7f5fd40d9640 1 librados: shutdown 2024-03-29T11:30:09.575 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:16.382 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:16.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.546+0000 7f1f3ee3e640 10 monclient: get_monmap_and_config 2024-03-29T11:30:16.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.546+0000 7f1f3ee3e640 10 monclient: build_initial_monmap 2024-03-29T11:30:16.539 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.546+0000 7f1f3ee3e640 10 monclient: monmap: 2024-03-29T11:30:16.539 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:16.539 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:16.539 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:30:16.547916+0000 2024-03-29T11:30:16.540 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:30:16.547916+0000 2024-03-29T11:30:16.540 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:16.540 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:16.540 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:16.540 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:30:16.540 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:30:16.540 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:16.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.548+0000 7f1f3ee3e640 1 Processor -- start 2024-03-29T11:30:16.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.548+0000 7f1f3ee3e640 1 -- start start 2024-03-29T11:30:16.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.548+0000 7f1f3ee3e640 10 monclient: init 2024-03-29T11:30:16.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.549+0000 7f1f3ee3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:16.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.549+0000 7f1f3ee3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:16.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.549+0000 7f1f3ee3e640 10 monclient(hunting): picked mon.noname-c con 0x7f1f381422b0 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:16.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.549+0000 7f1f3ee3e640 10 monclient(hunting): picked mon.noname-a con 0x7f1f38142690 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:16.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.549+0000 7f1f3ee3e640 10 monclient(hunting): picked mon.noname-b con 0x7f1f380b17d0 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:16.543 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.549+0000 7f1f3ee3e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f38142c80 con 0x7f1f38142690 2024-03-29T11:30:16.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.549+0000 7f1f3ee3e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380aae40 con 0x7f1f381422b0 2024-03-29T11:30:16.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.549+0000 7f1f3ee3e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3813fbd0 con 0x7f1f380b17d0 2024-03-29T11:30:16.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.549+0000 7f1f3ee3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:16.544 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.549+0000 7f1f3ee3e640 10 monclient(hunting): authenticate will time out at 1466.663878s 2024-03-29T11:30:16.546 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.553+0000 7f1f3cbda640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1f381422b0 0x7f1f3813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:46323/0 (socket says 172.21.15.59:46323) 2024-03-29T11:30:16.546 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.553+0000 7f1f3cbda640 1 -- 172.21.15.59:0/404337000 learned_addr learned my addr 172.21.15.59:0/404337000 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:16.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 1 -- 172.21.15.59:0/404337000 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285349669 0 0) 0x7f1f38142c80 con 0x7f1f38142690 2024-03-29T11:30:16.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:16.550 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:16.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:16.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 10 monclient(hunting): my global_id is 4605 2024-03-29T11:30:16.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 1 -- 172.21.15.59:0/404337000 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f20003520 con 0x7f1f38142690 2024-03-29T11:30:16.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 1 -- 172.21.15.59:0/404337000 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988125509 0 0) 0x7f1f380aae40 con 0x7f1f381422b0 2024-03-29T11:30:16.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:16.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:16.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:16.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 10 monclient(hunting): my global_id is 4615 2024-03-29T11:30:16.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 1 -- 172.21.15.59:0/404337000 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f38142c80 con 0x7f1f381422b0 2024-03-29T11:30:16.551 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 1 -- 172.21.15.59:0/404337000 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1414655494 0 0) 0x7f1f20003520 con 0x7f1f38142690 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 1 -- 172.21.15.59:0/404337000 >> v1:172.21.15.59:6790/0 conn(0x7f1f380b17d0 legacy=0x7f1f380b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 1 -- 172.21.15.59:0/404337000 >> v1:172.21.15.172:6789/0 conn(0x7f1f381422b0 legacy=0x7f1f3813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 1 monclient: found mon.noname-a 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 1 -- 172.21.15.59:0/404337000 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f38151940 con 0x7f1f38142690 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.557+0000 7f1f377fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 1 -- 172.21.15.59:0/404337000 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1f30001940 con 0x7f1f38142690 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.552 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 10 monclient: dump: 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 1 -- 172.21.15.59:0/404337000 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f30001b60 con 0x7f1f38142690 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 1 -- 172.21.15.59:0/404337000 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1f30001e70 con 0x7f1f38142690 2024-03-29T11:30:16.553 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.558+0000 7f1f377fe640 10 monclient: dump: 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:16.554 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:16.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.559+0000 7f1f3ee3e640 5 monclient: authenticate success, global_id 4605 2024-03-29T11:30:16.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.559+0000 7f1f3ee3e640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:16.555 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.559+0000 7f1f3ee3e640 4 set_mon_vals no callback set 2024-03-29T11:30:16.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.560+0000 7f1f3ee3e640 10 monclient: shutdown 2024-03-29T11:30:16.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.560+0000 7f1f3ee3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:16.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.560+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/404337000 >> v1:172.21.15.59:6789/0 conn(0x7f1f38142690 legacy=0x7f1f380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.561+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/404337000 shutdown_connections 2024-03-29T11:30:16.556 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.561+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/404337000 >> 172.21.15.59:0/404337000 conn(0x7f1f38066a00 msgr2=0x7f1f38066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:16.558 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.564+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/404337000 shutdown_connections 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.566+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/404337000 wait complete. 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.567+0000 7f1f3ee3e640 10 monclient: build_initial_monmap 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.567+0000 7f1f3ee3e640 10 monclient: monmap: 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:16.561 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.567+0000 7f1f3ee3e640 1 librados: starting msgr at 2024-03-29T11:30:16.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.567+0000 7f1f3ee3e640 1 librados: starting objecter 2024-03-29T11:30:16.562 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.567+0000 7f1f3ee3e640 1 Processor -- start 2024-03-29T11:30:16.569 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.571+0000 7f1f3ee3e640 1 -- start start 2024-03-29T11:30:16.569 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.571+0000 7f1f3ee3e640 1 librados: setting wanted keys 2024-03-29T11:30:16.569 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.571+0000 7f1f3ee3e640 1 librados: calling monclient init 2024-03-29T11:30:16.569 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.571+0000 7f1f3ee3e640 10 monclient: init 2024-03-29T11:30:16.569 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.572+0000 7f1f3ee3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:16.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.572+0000 7f1f3ee3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:16.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.573+0000 7f1f3ee3e640 10 monclient(hunting): picked mon.noname-c con 0x7f1f380b17d0 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:16.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.573+0000 7f1f3ee3e640 10 monclient(hunting): picked mon.noname-a con 0x7f1f381422b0 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:16.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.573+0000 7f1f3ee3e640 10 monclient(hunting): picked mon.noname-b con 0x7f1f38142690 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:16.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.573+0000 7f1f3ee3e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380acf30 con 0x7f1f381422b0 2024-03-29T11:30:16.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.573+0000 7f1f3ee3e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380ad160 con 0x7f1f38142690 2024-03-29T11:30:16.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.573+0000 7f1f3ee3e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f381e76b0 con 0x7f1f380b17d0 2024-03-29T11:30:16.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.573+0000 7f1f3ee3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:16.570 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.573+0000 7f1f3ee3e640 10 monclient(hunting): authenticate will time out at 1466.688433s 2024-03-29T11:30:16.571 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.578+0000 7f1f3d3db640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1f38142690 0x7f1f3813bbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.59:46455/0 (socket says 172.21.15.59:46455) 2024-03-29T11:30:16.571 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.578+0000 7f1f3d3db640 1 -- 172.21.15.59:0/724784244 learned_addr learned my addr 172.21.15.59:0/724784244 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:16.571 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.578+0000 7f1f357fa640 10 client.?.objecter ms_handle_connect 0x7f1f38142690 2024-03-29T11:30:16.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.578+0000 7f1f357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:16.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.578+0000 7f1f357fa640 10 client.?.objecter ms_handle_connect 0x7f1f381422b0 2024-03-29T11:30:16.572 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.578+0000 7f1f357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:16.581 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.581+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228169027 0 0) 0x7f1f380acf30 con 0x7f1f381422b0 2024-03-29T11:30:16.582 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.581+0000 7f1f357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:16.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.581+0000 7f1f357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:16.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.581+0000 7f1f357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:16.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.581+0000 7f1f357fa640 10 monclient(hunting): my global_id is 4608 2024-03-29T11:30:16.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.581+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f24004eb0 con 0x7f1f381422b0 2024-03-29T11:30:16.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.582+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911429584 0 0) 0x7f1f380ad160 con 0x7f1f38142690 2024-03-29T11:30:16.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.582+0000 7f1f357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:16.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.582+0000 7f1f357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:16.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.582+0000 7f1f357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:16.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.582+0000 7f1f357fa640 10 monclient(hunting): my global_id is 4618 2024-03-29T11:30:16.583 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.582+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f380acf30 con 0x7f1f38142690 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.582+0000 7f1f357fa640 10 client.?.objecter ms_handle_connect 0x7f1f380b17d0 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.582+0000 7f1f357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.582+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4023357831 0 0) 0x7f1f24004eb0 con 0x7f1f381422b0 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.582+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f380ad160 con 0x7f1f381422b0 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.582+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1f30001f30 con 0x7f1f381422b0 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 10 monclient(hunting): dump: 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.584 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112198443 0 0) 0x7f1f380acf30 con 0x7f1f38142690 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f24004eb0 con 0x7f1f38142690 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749470135 0 0) 0x7f1f381e76b0 con 0x7f1f380b17d0 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:30:16.585 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f380acf30 con 0x7f1f380b17d0 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1f2c001970 con 0x7f1f38142690 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.583+0000 7f1f357fa640 10 monclient(hunting): dump: 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:16.586 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.584+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2565405056 0 0) 0x7f1f380acf30 con 0x7f1f380b17d0 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.584+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f381e76b0 con 0x7f1f380b17d0 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.584+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1f28001830 con 0x7f1f380b17d0 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.584+0000 7f1f357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.584+0000 7f1f357fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.584+0000 7f1f357fa640 10 monclient(hunting): dump: 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:16.587 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.584+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.1 v1:172.21.15.172:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2100812520 0 0) 0x7f1f24004eb0 con 0x7f1f38142690 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.584+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 >> v1:172.21.15.59:6790/0 conn(0x7f1f380b17d0 legacy=0x7f1f38071a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.584+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 >> v1:172.21.15.59:6789/0 conn(0x7f1f381422b0 legacy=0x7f1f381e7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.584+0000 7f1f357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.584+0000 7f1f357fa640 1 monclient: found mon.b 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3806eee0 con 0x7f1f38142690 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f357fa640 10 monclient: _finish_auth 0 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f3ee3e640 5 monclient: authenticate success, global_id 4618 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f3ee3e640 10 monclient: _renew_subs 2024-03-29T11:30:16.588 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f3ee3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/724784244 --> v1:172.21.15.172:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f381e82e0 con 0x7f1f38142690 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f3ee3e640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f3ee3e640 10 monclient: _renew_subs 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f3ee3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.1 v1:172.21.15.172:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f2c001ad0 con 0x7f1f38142690 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.585+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/724784244 --> v1:172.21.15.172:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3813c300 con 0x7f1f38142690 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.586+0000 7f1f3ee3e640 1 librados: init done 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.586+0000 7f1f357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.586+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.1 v1:172.21.15.172:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f1f2c0038c0 con 0x7f1f38142690 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.586+0000 7f1f357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.586+0000 7f1f357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.586+0000 7f1f357fa640 10 monclient: dump: 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:16.589 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.586+0000 7f1f36ffd640 4 set_mon_vals no callback set 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.587+0000 7f1f3ee3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.587+0000 7f1f3ee3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.587+0000 7f1f3ee3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:30:16.590 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.587+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/724784244 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f381e7840 con 0x7f1f38142690 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.588+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.1 v1:172.21.15.172:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f1f2c003a20 con 0x7f1f38142690 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.589+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.1 v1:172.21.15.172:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2848875472 0 0) 0x7f1f2c084a20 con 0x7f1f38142690 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.589+0000 7f1f357fa640 10 client.4618.objecter ms_dispatch 0x7f1f38000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.589+0000 7f1f357fa640 3 client.4618.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.589+0000 7f1f357fa640 3 client.4618.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.589+0000 7f1f357fa640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.589+0000 7f1f3cbda640 10 monclient: get_auth_request con 0x7f1f2406e150 auth_method 0 2024-03-29T11:30:16.592 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.592+0000 7f1f357fa640 10 client.4618.objecter ms_handle_connect 0x7f1f2406e150 2024-03-29T11:30:16.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.599+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.1 v1:172.21.15.172:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f2c050130 con 0x7f1f38142690 2024-03-29T11:30:16.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.599+0000 7f1f357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:16.595 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.599+0000 7f1f357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:16.921 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.927+0000 7f1f3ee3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:16.921 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.927+0000 7f1f3ee3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:16.921 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.927+0000 7f1f3ee3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.172:6789/0 2024-03-29T11:30:16.921 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.927+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/724784244 --> v1:172.21.15.172:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f38075f10 con 0x7f1f38142690 2024-03-29T11:30:16.922 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.929+0000 7f1f357fa640 1 -- 172.21.15.59:0/724784244 <== mon.1 v1:172.21.15.172:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2144806415) 0x7f1f2c052230 con 0x7f1f38142690 2024-03-29T11:30:16.922 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.929+0000 7f1f357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:16.922 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.929+0000 7f1f357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:16.926 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:30:16.926 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":11,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:29:44.953085+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:44.953085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:16.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.935+0000 7f1f3ee3e640 10 librados: watch_flush enter 2024-03-29T11:30:16.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.935+0000 7f1f3ee3e640 10 librados: watch_flush exit 2024-03-29T11:30:16.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.935+0000 7f1f3ee3e640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-29T11:30:16.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.935+0000 7f1f3ee3e640 10 client.4618.objecter successfully canceled tick 2024-03-29T11:30:16.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.936+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/724784244 >> v1:172.21.15.172:6816/2420742773 conn(0x7f1f2406e150 legacy=0x7f1f24070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.936+0000 7f1f3ee3e640 10 monclient: shutdown 2024-03-29T11:30:16.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.936+0000 7f1f3ee3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:16.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.936+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/724784244 >> v1:172.21.15.172:6789/0 conn(0x7f1f38142690 legacy=0x7f1f3813bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.936+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/724784244 shutdown_connections 2024-03-29T11:30:16.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.936+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/724784244 >> 172.21.15.59:0/724784244 conn(0x7f1f38066a00 msgr2=0x7f1f38141720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:16.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.937+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/724784244 shutdown_connections 2024-03-29T11:30:16.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.938+0000 7f1f3ee3e640 1 -- 172.21.15.59:0/724784244 wait complete. 2024-03-29T11:30:16.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:16.940+0000 7f1f3ee3e640 1 librados: shutdown 2024-03-29T11:30:16.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:23.758 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:23.910 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.918+0000 7f8c4f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.918+0000 7f8c4f59e640 10 monclient: build_initial_monmap 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.918+0000 7f8c4f59e640 10 monclient: monmap: 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:30:23.918825+0000 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:30:23.918825+0000 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:30:23.911 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:23.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.918+0000 7f8c4f59e640 1 Processor -- start 2024-03-29T11:30:23.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.919+0000 7f8c4f59e640 1 -- start start 2024-03-29T11:30:23.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.919+0000 7f8c4f59e640 10 monclient: init 2024-03-29T11:30:23.912 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.919+0000 7f8c4f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:23.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.919+0000 7f8c4f59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:23.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.919+0000 7f8c4f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f8c500b2800 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:23.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.919+0000 7f8c4f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8c500a9c00 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:23.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.919+0000 7f8c4f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8c500b38c0 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:23.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.919+0000 7f8c4f59e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c50141eb0 con 0x7f8c500a9c00 2024-03-29T11:30:23.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4e59c640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f8c500b2800 0x7f8c50142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:40100/0 (socket says 172.21.15.59:40100) 2024-03-29T11:30:23.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4e59c640 1 -- 172.21.15.59:0/4107476356 learned_addr learned my addr 172.21.15.59:0/4107476356 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:23.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4f59e640 1 -- 172.21.15.59:0/4107476356 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c500b22d0 con 0x7f8c500b38c0 2024-03-29T11:30:23.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4f59e640 1 -- 172.21.15.59:0/4107476356 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c50142d30 con 0x7f8c500b2800 2024-03-29T11:30:23.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:23.913 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4f59e640 10 monclient(hunting): authenticate will time out at 1474.033870s 2024-03-29T11:30:23.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4dd9b640 1 -- 172.21.15.59:0/4107476356 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979973041 0 0) 0x7f8c50142d30 con 0x7f8c500b2800 2024-03-29T11:30:23.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:30:23.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:23.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:23.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4dd9b640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:30:23.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.920+0000 7f8c4dd9b640 1 -- 172.21.15.59:0/4107476356 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c3c003520 con 0x7f8c500b2800 2024-03-29T11:30:23.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.921+0000 7f8c4dd9b640 1 -- 172.21.15.59:0/4107476356 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1226577842 0 0) 0x7f8c3c003520 con 0x7f8c500b2800 2024-03-29T11:30:23.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.921+0000 7f8c4dd9b640 1 -- 172.21.15.59:0/4107476356 >> v1:172.21.15.172:6789/0 conn(0x7f8c500b38c0 legacy=0x7f8c5014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:23.914 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.921+0000 7f8c4dd9b640 1 -- 172.21.15.59:0/4107476356 >> v1:172.21.15.59:6789/0 conn(0x7f8c500a9c00 legacy=0x7f8c5013bb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:23.918 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:30:23.918 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 1 monclient: found mon.noname-b 2024-03-29T11:30:23.918 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:23.918 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:30:23.918 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 1 -- 172.21.15.59:0/4107476356 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c500aaf90 con 0x7f8c500b2800 2024-03-29T11:30:23.918 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:30:23.918 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:30:23.918 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:23.918 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 1 -- 172.21.15.59:0/4107476356 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8c40001690 con 0x7f8c500b2800 2024-03-29T11:30:23.918 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4dd9b640 10 monclient: dump: 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4f59e640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:30:23.919 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.924+0000 7f8c4f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.925+0000 7f8c4dd9b640 1 -- 172.21.15.59:0/4107476356 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c400017f0 con 0x7f8c500b2800 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.925+0000 7f8c4dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.925+0000 7f8c4dd9b640 1 -- 172.21.15.59:0/4107476356 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8c40001c40 con 0x7f8c500b2800 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.925+0000 7f8c4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.925+0000 7f8c4dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.925+0000 7f8c4dd9b640 10 monclient: dump: 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:23.920 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:23.921 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:23.921 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:23.921 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.925+0000 7f8c4f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:23.921 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.925+0000 7f8c4f59e640 4 set_mon_vals no callback set 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.926+0000 7f8c4f59e640 10 monclient: shutdown 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.926+0000 7f8c4f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.926+0000 7f8c4f59e640 1 -- 172.21.15.59:0/4107476356 >> v1:172.21.15.59:6790/0 conn(0x7f8c500b2800 legacy=0x7f8c50142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.926+0000 7f8c4f59e640 1 -- 172.21.15.59:0/4107476356 shutdown_connections 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.926+0000 7f8c4f59e640 1 -- 172.21.15.59:0/4107476356 >> 172.21.15.59:0/4107476356 conn(0x7f8c50066a00 msgr2=0x7f8c50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.926+0000 7f8c4f59e640 1 -- 172.21.15.59:0/4107476356 shutdown_connections 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.926+0000 7f8c4f59e640 1 -- 172.21.15.59:0/4107476356 wait complete. 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.927+0000 7f8c4f59e640 10 monclient: build_initial_monmap 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.927+0000 7f8c4f59e640 10 monclient: monmap: 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:30:23.923 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.928+0000 7f8c4f59e640 1 librados: starting msgr at 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.928+0000 7f8c4f59e640 1 librados: starting objecter 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.928+0000 7f8c4f59e640 1 Processor -- start 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.929+0000 7f8c4f59e640 1 -- start start 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.929+0000 7f8c4f59e640 1 librados: setting wanted keys 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.929+0000 7f8c4f59e640 1 librados: calling monclient init 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.929+0000 7f8c4f59e640 10 monclient: init 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c4f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c4f59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:23.924 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c4f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8c500a9c00 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c4f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8c500b2800 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c4f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f8c500b38c0 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c4f59e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5006f5b0 con 0x7f8c500a9c00 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c4f59e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c50070ed0 con 0x7f8c500b38c0 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c4f59e640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c500ab530 con 0x7f8c500b2800 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c4f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c4f59e640 10 monclient(hunting): authenticate will time out at 1474.043889s 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c45d9b640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f8c500b2800 0x7f8c5006eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:41088/0 (socket says 172.21.15.59:41088) 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c45d9b640 1 -- 172.21.15.59:0/2827370824 learned_addr learned my addr 172.21.15.59:0/2827370824 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c477fe640 10 client.?.objecter ms_handle_connect 0x7f8c500b2800 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286991771 0 0) 0x7f8c500ab530 con 0x7f8c500b2800 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:23.925 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:23.926 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:23.926 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c477fe640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:30:23.926 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c30004eb0 con 0x7f8c500b2800 2024-03-29T11:30:23.926 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c477fe640 10 client.?.objecter ms_handle_connect 0x7f8c500b38c0 2024-03-29T11:30:23.926 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.930+0000 7f8c477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:23.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3052202470 0 0) 0x7f8c30004eb0 con 0x7f8c500b2800 2024-03-29T11:30:23.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c500ab530 con 0x7f8c500b2800 2024-03-29T11:30:23.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953968165 0 0) 0x7f8c50070ed0 con 0x7f8c500b38c0 2024-03-29T11:30:23.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:23.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:23.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:23.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 10 monclient(hunting): my global_id is 4621 2024-03-29T11:30:23.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c30004eb0 con 0x7f8c500b38c0 2024-03-29T11:30:23.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8c38001930 con 0x7f8c500b2800 2024-03-29T11:30:23.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:23.929 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 10 monclient(hunting): dump: 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:23.930 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 673109925 0 0) 0x7f8c30004eb0 con 0x7f8c500b38c0 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c50070ed0 con 0x7f8c500b38c0 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1523123658 0 0) 0x7f8c500ab530 con 0x7f8c500b2800 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 >> v1:172.21.15.172:6789/0 conn(0x7f8c500b38c0 legacy=0x7f8c501e6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.931+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 >> v1:172.21.15.59:6789/0 conn(0x7f8c500a9c00 legacy=0x7f8c5013ab40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c477fe640 1 monclient: found mon.c 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c477fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c50070a80 con 0x7f8c500b2800 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c477fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c4f59e640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c4f59e640 10 monclient: _renew_subs 2024-03-29T11:30:23.931 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c4f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c4f59e640 1 -- 172.21.15.59:0/2827370824 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c50070340 con 0x7f8c500b2800 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c4f59e640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c4f59e640 10 monclient: _renew_subs 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c4f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c4f59e640 1 -- 172.21.15.59:0/2827370824 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c50071400 con 0x7f8c500b2800 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.932+0000 7f8c4f59e640 1 librados: init done 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.933+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c38001d40 con 0x7f8c500b2800 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.933+0000 7f8c477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.933+0000 7f8c4d59a640 4 set_mon_vals no callback set 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.933+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f8c380038a0 con 0x7f8c500b2800 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.933+0000 7f8c477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.933+0000 7f8c477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.933+0000 7f8c477fe640 10 monclient: dump: 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:23.932 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.935+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f8c38003a00 con 0x7f8c500b2800 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.935+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2848875472 0 0) 0x7f8c380849e0 con 0x7f8c500b2800 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.935+0000 7f8c477fe640 10 client.4631.objecter ms_dispatch 0x7f8c50000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.935+0000 7f8c477fe640 3 client.4631.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.935+0000 7f8c477fe640 3 client.4631.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.935+0000 7f8c4e59c640 10 monclient: get_auth_request con 0x7f8c3006dfa0 auth_method 0 2024-03-29T11:30:23.933 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.935+0000 7f8c4f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:23.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.935+0000 7f8c4f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:23.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.935+0000 7f8c4f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:23.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.936+0000 7f8c4f59e640 1 -- 172.21.15.59:0/2827370824 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c34005080 con 0x7f8c500b2800 2024-03-29T11:30:23.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.936+0000 7f8c477fe640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:30:23.934 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.936+0000 7f8c477fe640 10 client.4631.objecter ms_handle_connect 0x7f8c3006dfa0 2024-03-29T11:30:23.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.943+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c380500f0 con 0x7f8c500b2800 2024-03-29T11:30:23.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.943+0000 7f8c477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:23.936 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:23.943+0000 7f8c477fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:24.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.272+0000 7f8c4f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:24.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.272+0000 7f8c4f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:24.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.272+0000 7f8c4f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:24.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.272+0000 7f8c4f59e640 1 -- 172.21.15.59:0/2827370824 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c34005330 con 0x7f8c500b2800 2024-03-29T11:30:24.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.273+0000 7f8c477fe640 1 -- 172.21.15.59:0/2827370824 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2144806415) 0x7f8c380521f0 con 0x7f8c500b2800 2024-03-29T11:30:24.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.273+0000 7f8c477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:24.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.273+0000 7f8c477fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:24.265 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:30:24.266 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":11,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:29:44.953085+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:44.953085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:24.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 10 librados: watch_flush enter 2024-03-29T11:30:24.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 10 librados: watch_flush exit 2024-03-29T11:30:24.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:30:24.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:30:24.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 1 -- 172.21.15.59:0/2827370824 >> v1:172.21.15.172:6816/2420742773 conn(0x7f8c3006dfa0 legacy=0x7f8c30070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 10 monclient: shutdown 2024-03-29T11:30:24.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:24.269 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 1 -- 172.21.15.59:0/2827370824 >> v1:172.21.15.59:6790/0 conn(0x7f8c500b2800 legacy=0x7f8c5006eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 1 -- 172.21.15.59:0/2827370824 shutdown_connections 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 1 -- 172.21.15.59:0/2827370824 >> 172.21.15.59:0/2827370824 conn(0x7f8c50066a00 msgr2=0x7f8c501403e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 1 -- 172.21.15.59:0/2827370824 shutdown_connections 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 1 -- 172.21.15.59:0/2827370824 wait complete. 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:24.276+0000 7f8c4f59e640 1 librados: shutdown 2024-03-29T11:30:24.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:31.089 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:31.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.241+0000 7f9b22c25640 10 monclient: get_monmap_and_config 2024-03-29T11:30:31.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.241+0000 7f9b22c25640 10 monclient: build_initial_monmap 2024-03-29T11:30:31.234 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.241+0000 7f9b22c25640 10 monclient: monmap: 2024-03-29T11:30:31.234 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:31.234 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:31.234 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:30:31.242229+0000 2024-03-29T11:30:31.234 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:30:31.242229+0000 2024-03-29T11:30:31.234 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:31.235 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:31.235 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:31.235 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:30:31.235 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:30:31.235 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:31.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.242+0000 7f9b22c25640 1 Processor -- start 2024-03-29T11:30:31.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.242+0000 7f9b22c25640 1 -- start start 2024-03-29T11:30:31.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.242+0000 7f9b22c25640 10 monclient: init 2024-03-29T11:30:31.236 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.243+0000 7f9b22c25640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:31.239 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.243+0000 7f9b22c25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.243+0000 7f9b22c25640 10 monclient(hunting): picked mon.noname-c con 0x7f9b1c1457f0 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.243+0000 7f9b22c25640 10 monclient(hunting): picked mon.noname-b con 0x7f9b1c13f650 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.243+0000 7f9b22c25640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1c0a9d70 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.243+0000 7f9b22c25640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1c0aaef0 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.243+0000 7f9b22c25640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1c145bd0 con 0x7f9b1c1457f0 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.243+0000 7f9b22c25640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1c145eb0 con 0x7f9b1c13f650 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.243+0000 7f9b22c25640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.243+0000 7f9b22c25640 10 monclient(hunting): authenticate will time out at 1481.357615s 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.244+0000 7f9b211c2640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9b1c0a9d70 0x7f9b1c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50408/0 (socket says 172.21.15.59:50408) 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.244+0000 7f9b211c2640 1 -- 172.21.15.59:0/550637857 learned_addr learned my addr 172.21.15.59:0/550637857 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.244+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/550637857 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495716490 0 0) 0x7f9b1c0aaef0 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.244+0000 7f9b1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.240 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.244+0000 7f9b1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.244+0000 7f9b1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.244+0000 7f9b1b7fe640 10 monclient(hunting): my global_id is 4611 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.244+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/550637857 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b0c003520 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/550637857 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3346194348 0 0) 0x7f9b0c003520 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/550637857 >> v1:172.21.15.59:6790/0 conn(0x7f9b1c13f650 legacy=0x7f9b1c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/550637857 >> v1:172.21.15.172:6789/0 conn(0x7f9b1c1457f0 legacy=0x7f9b1c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.59:6789/0 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/550637857 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c146340 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:31.241 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/550637857 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9b14001a10 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 10 monclient: dump: 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:31.242 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:31.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b22c25640 5 monclient: authenticate success, global_id 4611 2024-03-29T11:30:31.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b22c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:31.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/550637857 <== mon.0 v1:172.21.15.59:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b14001be0 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:31.243 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b22c25640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b22c25640 4 set_mon_vals no callback set 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/550637857 <== mon.0 v1:172.21.15.59:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9b14001ff0 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.245+0000 7f9b1b7fe640 10 monclient: dump: 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:31.244 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:31.245 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:31.245 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:31.245 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:31.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.246+0000 7f9b22c25640 10 monclient: shutdown 2024-03-29T11:30:31.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.246+0000 7f9b22c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:31.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.246+0000 7f9b22c25640 1 -- 172.21.15.59:0/550637857 >> v1:172.21.15.59:6789/0 conn(0x7f9b1c0a9d70 legacy=0x7f9b1c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.246+0000 7f9b22c25640 1 -- 172.21.15.59:0/550637857 shutdown_connections 2024-03-29T11:30:31.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.246+0000 7f9b22c25640 1 -- 172.21.15.59:0/550637857 >> 172.21.15.59:0/550637857 conn(0x7f9b1c066a00 msgr2=0x7f9b1c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:31.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.246+0000 7f9b22c25640 1 -- 172.21.15.59:0/550637857 shutdown_connections 2024-03-29T11:30:31.245 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.247+0000 7f9b22c25640 1 -- 172.21.15.59:0/550637857 wait complete. 2024-03-29T11:30:31.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.248+0000 7f9b22c25640 10 monclient: build_initial_monmap 2024-03-29T11:30:31.252 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.248+0000 7f9b22c25640 10 monclient: monmap: 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.248+0000 7f9b22c25640 1 librados: starting msgr at 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.248+0000 7f9b22c25640 1 librados: starting objecter 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.248+0000 7f9b22c25640 1 Processor -- start 2024-03-29T11:30:31.253 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.249+0000 7f9b22c25640 1 -- start start 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.249+0000 7f9b22c25640 1 librados: setting wanted keys 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.249+0000 7f9b22c25640 1 librados: calling monclient init 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.249+0000 7f9b22c25640 10 monclient: init 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.250+0000 7f9b22c25640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.250+0000 7f9b22c25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.250+0000 7f9b22c25640 10 monclient(hunting): picked mon.noname-b con 0x7f9b1c0a9d70 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.251+0000 7f9b22c25640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1c13f650 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.251+0000 7f9b22c25640 10 monclient(hunting): picked mon.noname-c con 0x7f9b1c1457f0 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.251+0000 7f9b211c2640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f9b1c1457f0 0x7f9b1c1e1ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:42120/0 (socket says 172.21.15.59:42120) 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.251+0000 7f9b211c2640 1 -- 172.21.15.59:0/2207629250 learned_addr learned my addr 172.21.15.59:0/2207629250 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.251+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1c142cd0 con 0x7f9b1c13f650 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.251+0000 7f9b197fa640 10 client.?.objecter ms_handle_connect 0x7f9b1c13f650 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.251+0000 7f9b197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:31.254 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.251+0000 7f9b197fa640 10 client.?.objecter ms_handle_connect 0x7f9b1c1457f0 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.251+0000 7f9b197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.251+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1c141ec0 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1c1d7900 con 0x7f9b1c1457f0 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 client.?.objecter ms_handle_connect 0x7f9b1c0a9d70 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b22c25640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b22c25640 10 monclient(hunting): authenticate will time out at 1481.365827s 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088589786 0 0) 0x7f9b1c141ec0 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): my global_id is 4624 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b00004eb0 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441522303 0 0) 0x7f9b1c1d7900 con 0x7f9b1c1457f0 2024-03-29T11:30:31.255 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:30:31.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b1c141ec0 con 0x7f9b1c1457f0 2024-03-29T11:30:31.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676524626 0 0) 0x7f9b1c142cd0 con 0x7f9b1c13f650 2024-03-29T11:30:31.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 10 monclient(hunting): my global_id is 4614 2024-03-29T11:30:31.256 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.252+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b1c1d7900 con 0x7f9b1c13f650 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2997118509 0 0) 0x7f9b1c141ec0 con 0x7f9b1c1457f0 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b1c142cd0 con 0x7f9b1c1457f0 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9b14001ef0 con 0x7f9b1c1457f0 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 10 monclient(hunting): dump: 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:31.257 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4074500529 0 0) 0x7f9b1c1d7900 con 0x7f9b1c13f650 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.59:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b1c141ec0 con 0x7f9b1c13f650 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9b08001830 con 0x7f9b1c13f650 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 10 monclient(hunting): dump: 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.258 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678870055 0 0) 0x7f9b00004eb0 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.172:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b1c1d7900 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9b10001750 con 0x7f9b1c0a9d70 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.253+0000 7f9b197fa640 10 monclient(hunting): dump: 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:31.259 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 410314848 0 0) 0x7f9b1c142cd0 con 0x7f9b1c1457f0 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 >> v1:172.21.15.172:6789/0 conn(0x7f9b1c0a9d70 legacy=0x7f9b1c1d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 >> v1:172.21.15.59:6789/0 conn(0x7f9b1c13f650 legacy=0x7f9b1c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b197fa640 10 monclient: _finish_hunting 0 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b197fa640 1 monclient: found mon.c 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:31.260 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c1d7470 con 0x7f9b1c1457f0 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b197fa640 10 monclient: _finish_auth 0 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b197fa640 10 monclient: _check_auth_tickets 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b22c25640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b22c25640 10 monclient: _renew_subs 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b22c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b1c140db0 con 0x7f9b1c1457f0 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b22c25640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b22c25640 10 monclient: _renew_subs 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b22c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b1c1452e0 con 0x7f9b1c1457f0 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.254+0000 7f9b22c25640 1 librados: init done 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.255+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b14002050 con 0x7f9b1c1457f0 2024-03-29T11:30:31.261 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.255+0000 7f9b197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:31.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.255+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f9b14005110 con 0x7f9b1c1457f0 2024-03-29T11:30:31.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.255+0000 7f9b197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.255+0000 7f9b197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:31.262 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.255+0000 7f9b197fa640 10 monclient: dump: 2024-03-29T11:30:31.262 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:31.262 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:31.262 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.262 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:31.262 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.262 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.255+0000 7f9b1affd640 4 set_mon_vals no callback set 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.256+0000 7f9b22c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.256+0000 7f9b22c25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.256+0000 7f9b22c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.256+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9afc005080 con 0x7f9b1c1457f0 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.258+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f9b1401b0e0 con 0x7f9b1c1457f0 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.258+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2848875472 0 0) 0x7f9b14084df0 con 0x7f9b1c1457f0 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.259+0000 7f9b197fa640 10 client.4637.objecter ms_dispatch 0x7f9b1c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:31.263 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.259+0000 7f9b197fa640 3 client.4637.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:31.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.259+0000 7f9b197fa640 3 client.4637.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:31.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.259+0000 7f9b197fa640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T11:30:31.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.259+0000 7f9b209c1640 10 monclient: get_auth_request con 0x7f9b0006e0d0 auth_method 0 2024-03-29T11:30:31.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.259+0000 7f9b197fa640 10 client.4637.objecter ms_handle_connect 0x7f9b0006e0d0 2024-03-29T11:30:31.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.265+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b140506d0 con 0x7f9b1c1457f0 2024-03-29T11:30:31.264 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.265+0000 7f9b197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:31.265 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.265+0000 7f9b197fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:31.631 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.638+0000 7f9b22c25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:31.632 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.638+0000 7f9b22c25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:31.632 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.638+0000 7f9b22c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:31.632 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.638+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9afc005330 con 0x7f9b1c1457f0 2024-03-29T11:30:31.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.640+0000 7f9b197fa640 1 -- 172.21.15.59:0/2207629250 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2144806415) 0x7f9b14052600 con 0x7f9b1c1457f0 2024-03-29T11:30:31.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.640+0000 7f9b197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:31.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.640+0000 7f9b197fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:31.634 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:30:31.634 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":11,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:29:44.953085+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:44.953085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:31.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 10 librados: watch_flush enter 2024-03-29T11:30:31.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 10 librados: watch_flush exit 2024-03-29T11:30:31.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T11:30:31.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 10 client.4637.objecter successfully canceled tick 2024-03-29T11:30:31.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 >> v1:172.21.15.172:6816/2420742773 conn(0x7f9b0006e0d0 legacy=0x7f9b00070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 10 monclient: shutdown 2024-03-29T11:30:31.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:31.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 >> v1:172.21.15.59:6790/0 conn(0x7f9b1c1457f0 legacy=0x7f9b1c1e1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 shutdown_connections 2024-03-29T11:30:31.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 >> 172.21.15.59:0/2207629250 conn(0x7f9b1c066a00 msgr2=0x7f9b1c13b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:31.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 shutdown_connections 2024-03-29T11:30:31.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.644+0000 7f9b22c25640 1 -- 172.21.15.59:0/2207629250 wait complete. 2024-03-29T11:30:31.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:31.645+0000 7f9b22c25640 1 librados: shutdown 2024-03-29T11:30:31.652 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:38.459 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:38.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.619+0000 7f25b59cf640 10 monclient: get_monmap_and_config 2024-03-29T11:30:38.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.619+0000 7f25b59cf640 10 monclient: build_initial_monmap 2024-03-29T11:30:38.612 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.619+0000 7f25b59cf640 10 monclient: monmap: 2024-03-29T11:30:38.612 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:38.612 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:38.612 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:30:38.620183+0000 2024-03-29T11:30:38.612 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:30:38.620183+0000 2024-03-29T11:30:38.612 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:38.612 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:38.613 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:38.613 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-c 2024-03-29T11:30:38.613 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-b 2024-03-29T11:30:38.613 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:38.614 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.621+0000 7f25b59cf640 1 Processor -- start 2024-03-29T11:30:38.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.622+0000 7f25b59cf640 1 -- start start 2024-03-29T11:30:38.615 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.622+0000 7f25b59cf640 10 monclient: init 2024-03-29T11:30:38.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.623+0000 7f25b59cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:38.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.623+0000 7f25b59cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:38.618 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.623+0000 7f25b59cf640 10 monclient(hunting): picked mon.noname-c con 0x7f25b006ea10 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:38.619 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.626+0000 7f25b59cf640 10 monclient(hunting): picked mon.noname-b con 0x7f25b013bd30 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:38.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.626+0000 7f25b59cf640 10 monclient(hunting): picked mon.noname-a con 0x7f25b006edf0 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:38.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.627+0000 7f25b59cf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b0070f40 con 0x7f25b006edf0 2024-03-29T11:30:38.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.627+0000 7f25b59cf640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b0071360 con 0x7f25b006ea10 2024-03-29T11:30:38.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.627+0000 7f25b59cf640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b0072e60 con 0x7f25b013bd30 2024-03-29T11:30:38.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.627+0000 7f25b59cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:38.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.627+0000 7f25b59cf640 10 monclient(hunting): authenticate will time out at 1488.740909s 2024-03-29T11:30:38.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.627+0000 7f25a7fff640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f25b013bd30 0x7f25b013e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:46052/0 (socket says 172.21.15.59:46052) 2024-03-29T11:30:38.620 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.627+0000 7f25a7fff640 1 -- 172.21.15.59:0/571714860 learned_addr learned my addr 172.21.15.59:0/571714860 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:38.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 1 -- 172.21.15.59:0/571714860 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629265856 0 0) 0x7f25b0071360 con 0x7f25b006ea10 2024-03-29T11:30:38.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:38.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:38.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:38.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 10 monclient(hunting): my global_id is 4627 2024-03-29T11:30:38.622 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 1 -- 172.21.15.59:0/571714860 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259c004630 con 0x7f25b006ea10 2024-03-29T11:30:38.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 1 -- 172.21.15.59:0/571714860 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334817503 0 0) 0x7f25b0072e60 con 0x7f25b013bd30 2024-03-29T11:30:38.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:38.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:38.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:38.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:30:38.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 1 -- 172.21.15.59:0/571714860 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259c004950 con 0x7f25b013bd30 2024-03-29T11:30:38.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 1 -- 172.21.15.59:0/571714860 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2972421583 0 0) 0x7f259c004950 con 0x7f25b013bd30 2024-03-29T11:30:38.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 1 -- 172.21.15.59:0/571714860 >> v1:172.21.15.172:6789/0 conn(0x7f25b006ea10 legacy=0x7f25b013b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.623 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.628+0000 7f25ae7fc640 1 -- 172.21.15.59:0/571714860 >> v1:172.21.15.59:6789/0 conn(0x7f25b006edf0 legacy=0x7f25b0149d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:38.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:38.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 1 monclient: found mon.noname-b 2024-03-29T11:30:38.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:38.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.59:6790/0 2024-03-29T11:30:38.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 1 -- 172.21.15.59:0/571714860 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b006fb00 con 0x7f25b013bd30 2024-03-29T11:30:38.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:38.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:38.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:38.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 1 -- 172.21.15.59:0/571714860 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2598001670 con 0x7f25b013bd30 2024-03-29T11:30:38.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:38.626 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.629+0000 7f25ae7fc640 10 monclient: dump: 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.630+0000 7f25ae7fc640 1 -- 172.21.15.59:0/571714860 <== mon.2 v1:172.21.15.59:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2598001850 con 0x7f25b013bd30 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.630+0000 7f25ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.630+0000 7f25ae7fc640 1 -- 172.21.15.59:0/571714860 <== mon.2 v1:172.21.15.59:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f2598001c60 con 0x7f25b013bd30 2024-03-29T11:30:38.627 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.630+0000 7f25ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.630+0000 7f25ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.630+0000 7f25ae7fc640 10 monclient: dump: 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.631+0000 7f25b59cf640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.631+0000 7f25b59cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:38.628 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.631+0000 7f25b59cf640 4 set_mon_vals no callback set 2024-03-29T11:30:38.629 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.632+0000 7f25b59cf640 10 monclient: shutdown 2024-03-29T11:30:38.629 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.632+0000 7f25b59cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:38.629 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.632+0000 7f25b59cf640 1 -- 172.21.15.59:0/571714860 >> v1:172.21.15.59:6790/0 conn(0x7f25b013bd30 legacy=0x7f25b013e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.629 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.632+0000 7f25b59cf640 1 -- 172.21.15.59:0/571714860 shutdown_connections 2024-03-29T11:30:38.629 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.632+0000 7f25b59cf640 1 -- 172.21.15.59:0/571714860 >> 172.21.15.59:0/571714860 conn(0x7f25b00b3f40 msgr2=0x7f25b00b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:38.629 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.633+0000 7f25b59cf640 1 -- 172.21.15.59:0/571714860 shutdown_connections 2024-03-29T11:30:38.629 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.633+0000 7f25b59cf640 1 -- 172.21.15.59:0/571714860 wait complete. 2024-03-29T11:30:38.631 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.634+0000 7f25b59cf640 10 monclient: build_initial_monmap 2024-03-29T11:30:38.631 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.634+0000 7f25b59cf640 10 monclient: monmap: 2024-03-29T11:30:38.631 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.noname-b 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.noname-c 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.634+0000 7f25b59cf640 1 librados: starting msgr at 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.634+0000 7f25b59cf640 1 librados: starting objecter 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.634+0000 7f25b59cf640 1 Processor -- start 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.634+0000 7f25b59cf640 1 -- start start 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.634+0000 7f25b59cf640 1 librados: setting wanted keys 2024-03-29T11:30:38.632 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.634+0000 7f25b59cf640 1 librados: calling monclient init 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.634+0000 7f25b59cf640 10 monclient: init 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.636+0000 7f25b59cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.636+0000 7f25b59cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.637+0000 7f25b59cf640 10 monclient(hunting): picked mon.noname-c con 0x7f25b006ea10 addr v1:172.21.15.59:6790/0 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.637+0000 7f25b59cf640 10 monclient(hunting): picked mon.noname-a con 0x7f25b006edf0 addr v1:172.21.15.59:6789/0 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.637+0000 7f25b59cf640 10 monclient(hunting): picked mon.noname-b con 0x7f25b013bd30 addr v1:172.21.15.172:6789/0 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.637+0000 7f25b59cf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b013c8b0 con 0x7f25b006edf0 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.637+0000 7f25b59cf640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b013c390 con 0x7f25b013bd30 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.637+0000 7f25b59cf640 1 -- --> v1:172.21.15.59:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b0142740 con 0x7f25b006ea10 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.637+0000 7f25b59cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.637+0000 7f25b59cf640 10 monclient(hunting): authenticate will time out at 1488.750919s 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.638+0000 7f25aeffd640 1 --1- >> v1:172.21.15.59:6790/0 conn(0x7f25b006ea10 0x7f25b013dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.59:6790/0 says I am v1:172.21.15.59:50380/0 (socket says 172.21.15.59:50380) 2024-03-29T11:30:38.633 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.638+0000 7f25aeffd640 1 -- 172.21.15.59:0/3087942110 learned_addr learned my addr 172.21.15.59:0/3087942110 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-29T11:30:38.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.640+0000 7f25acff9640 10 client.?.objecter ms_handle_connect 0x7f25b006ea10 2024-03-29T11:30:38.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.640+0000 7f25acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:38.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.640+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 <== mon.2 v1:172.21.15.59:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205161692 0 0) 0x7f25b0142740 con 0x7f25b006ea10 2024-03-29T11:30:38.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.640+0000 7f25acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:38.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.640+0000 7f25acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:38.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.640+0000 7f25acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:38.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.640+0000 7f25acff9640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:30:38.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.641+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 --> v1:172.21.15.59:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590004eb0 con 0x7f25b006ea10 2024-03-29T11:30:38.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.641+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 <== mon.2 v1:172.21.15.59:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1092987611 0 0) 0x7f2590004eb0 con 0x7f25b006ea10 2024-03-29T11:30:38.638 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.641+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 --> v1:172.21.15.59:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25b0142740 con 0x7f25b006ea10 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.641+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 <== mon.2 v1:172.21.15.59:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f25a00019f0 con 0x7f25b006ea10 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.641+0000 7f25acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.641+0000 7f25acff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.641+0000 7f25acff9640 10 monclient(hunting): dump: 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:38.639 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.641+0000 7f25acff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 <== mon.2 v1:172.21.15.59:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3148443504 0 0) 0x7f25b0142740 con 0x7f25b006ea10 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 >> v1:172.21.15.172:6789/0 conn(0x7f25b013bd30 legacy=0x7f25b01db3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 >> v1:172.21.15.59:6789/0 conn(0x7f25b006edf0 legacy=0x7f25b01368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25acff9640 10 monclient: _finish_hunting 0 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25acff9640 1 monclient: found mon.c 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25acff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 --> v1:172.21.15.59:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b0136fb0 con 0x7f25b006ea10 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25acff9640 10 monclient: _finish_auth 0 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25acff9640 10 monclient: _check_auth_tickets 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25b59cf640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25b59cf640 10 monclient: _renew_subs 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25b59cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:38.640 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25b59cf640 1 -- 172.21.15.59:0/3087942110 --> v1:172.21.15.59:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b0137260 con 0x7f25b006ea10 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25b59cf640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25b59cf640 10 monclient: _renew_subs 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25b59cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25b59cf640 1 -- 172.21.15.59:0/3087942110 --> v1:172.21.15.59:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b01cd5a0 con 0x7f25b006ea10 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.642+0000 7f25b59cf640 1 librados: init done 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.643+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 <== mon.2 v1:172.21.15.59:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25a0001d10 con 0x7f25b006ea10 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.643+0000 7f25acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.643+0000 7f25a5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.643+0000 7f25adffb640 4 set_mon_vals no callback set 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.644+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 <== mon.2 v1:172.21.15.59:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4292408440 0 0) 0x7f25a0003a40 con 0x7f25b006ea10 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.644+0000 7f25acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.644+0000 7f25acff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.644+0000 7f25acff9640 10 monclient: dump: 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2024-03-29T11:30:38.641 INFO:teuthology.misc.health.smithi059.stderr:fsid 6661e61e-58ea-43e3-8b64-266f0a1dca60 2024-03-29T11:30:38.642 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:38.642 INFO:teuthology.misc.health.smithi059.stderr:created 2024-03-29T11:19:01.913054+0000 2024-03-29T11:30:38.642 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:38.642 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2024-03-29T11:30:38.642 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.a 2024-03-29T11:30:38.642 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.172:6789/0 mon.b 2024-03-29T11:30:38.642 INFO:teuthology.misc.health.smithi059.stderr:2: v1:172.21.15.59:6790/0 mon.c 2024-03-29T11:30:38.642 INFO:teuthology.misc.health.smithi059.stderr: 2024-03-29T11:30:38.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.646+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 <== mon.2 v1:172.21.15.59:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2695313187 0 0) 0x7f25a0003ba0 con 0x7f25b006ea10 2024-03-29T11:30:38.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.646+0000 7f25a5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:38.643 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.646+0000 7f25a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:38.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.646+0000 7f25a5ffb640 1 -- 172.21.15.59:0/3087942110 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2594005080 con 0x7f25b006ea10 2024-03-29T11:30:38.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.647+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 <== mon.2 v1:172.21.15.59:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2848875472 0 0) 0x7f25a0084b00 con 0x7f25b006ea10 2024-03-29T11:30:38.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.647+0000 7f25acff9640 10 client.4643.objecter ms_dispatch 0x7f25b0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:38.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.647+0000 7f25acff9640 3 client.4643.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:38.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.647+0000 7f25acff9640 3 client.4643.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:38.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.647+0000 7f25acff9640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T11:30:38.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.647+0000 7f25a7fff640 10 monclient: get_auth_request con 0x7f259006dc90 auth_method 0 2024-03-29T11:30:38.644 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.647+0000 7f25acff9640 10 client.4643.objecter ms_handle_connect 0x7f259006dc90 2024-03-29T11:30:38.648 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.656+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 <== mon.2 v1:172.21.15.59:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25a0050210 con 0x7f25b006ea10 2024-03-29T11:30:38.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.656+0000 7f25acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:38.649 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.656+0000 7f25acff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:38.989 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.996+0000 7f25a5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:38.989 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.996+0000 7f25a5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:38.989 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.996+0000 7f25a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.59:6790/0 2024-03-29T11:30:38.989 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.996+0000 7f25a5ffb640 1 -- 172.21.15.59:0/3087942110 --> v1:172.21.15.59:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2594005330 con 0x7f25b006ea10 2024-03-29T11:30:38.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.997+0000 7f25acff9640 1 -- 172.21.15.59:0/3087942110 <== mon.2 v1:172.21.15.59:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2144806415) 0x7f25a0052310 con 0x7f25b006ea10 2024-03-29T11:30:38.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.997+0000 7f25acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:38.990 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:38.997+0000 7f25acff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:38.990 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-29T11:30:38.990 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":11,"fsid":"6661e61e-58ea-43e3-8b64-266f0a1dca60","created":"2024-03-29T11:19:40.025248+0000","modified":"2024-03-29T11:29:44.953085+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:44.953085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"521f85c6-3db6-4c87-b0e3-e566a02771c1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"a3fb41ae-75c4-43a8-ae52-b40aa5699776","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9314facb-46cf-4601-b836-0471cf393809","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"27bf268d-5153-49bb-af84-46d68b59c5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a26fa133-b0ec-42d4-b9ba-d6150e5b456b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c907b48d-1121-4419-9221-28db3efd1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5efd8b3d-3ac6-48ee-9986-dc6dc3b6c8e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee31832e-2287-4b25-b133-3af8315e4497","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.000+0000 7f25a5ffb640 10 librados: watch_flush enter 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.000+0000 7f25a5ffb640 10 librados: watch_flush exit 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.000+0000 7f25a5ffb640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.000+0000 7f25a5ffb640 10 client.4643.objecter successfully canceled tick 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.001+0000 7f25a5ffb640 1 -- 172.21.15.59:0/3087942110 >> v1:172.21.15.172:6816/2420742773 conn(0x7f259006dc90 legacy=0x7f2590070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.001+0000 7f25a5ffb640 10 monclient: shutdown 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.001+0000 7f25a5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.001+0000 7f25a5ffb640 1 -- 172.21.15.59:0/3087942110 >> v1:172.21.15.59:6790/0 conn(0x7f25b006ea10 legacy=0x7f25b013dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.001+0000 7f25a5ffb640 1 -- 172.21.15.59:0/3087942110 shutdown_connections 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.001+0000 7f25a5ffb640 1 -- 172.21.15.59:0/3087942110 >> 172.21.15.59:0/3087942110 conn(0x7f25b00b3f40 msgr2=0x7f25b013b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.001+0000 7f25a5ffb640 1 -- 172.21.15.59:0/3087942110 shutdown_connections 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.001+0000 7f25a5ffb640 1 -- 172.21.15.59:0/3087942110 wait complete. 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi059.stderr:2024-03-29T11:30:39.001+0000 7f25a5ffb640 1 librados: shutdown 2024-03-29T11:30:39.008 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:39.008 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:30:39.010 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:30:39.010 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:30:39.010 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.045 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:30:39.045 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:30:39.045 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.124 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:30:39.125 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:30:39.125 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.198 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:30:39.198 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:30:39.198 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.269 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:30:39.270 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:30:39.270 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.307 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:30:39.307 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:30:39.307 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.384 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:30:39.384 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:30:39.385 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.460 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:30:39.460 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:30:39.461 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.527 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:30:39.527 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:30:39.527 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:30:39.527 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.551 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:30:39.551 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:30:39.551 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.592 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:30:39.593 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:30:39.593 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:30:39.593 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.604 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:30:39.604 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:30:39.604 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.656 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:30:39.656 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:30:39.656 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:39.667 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:30:39.667 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:30:39.667 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:30:39.728 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-29T11:30:39.728 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:30:42.541 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-29T11:30:42.541 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:30:42.653 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-29T11:30:42.653 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:30:42.721 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-29T11:30:42.722 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:30:42.787 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-29T11:30:42.787 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:30:44.116 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-29T11:30:44.116 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:30:44.179 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-29T11:30:44.179 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:30:44.244 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-29T11:30:44.244 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:30:44.302 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:30:44.304 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689/data/mon.a.tgz 2024-03-29T11:30:44.304 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-03-29T11:30:44.322 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.pcNfQyJStp 2024-03-29T11:30:44.323 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.pcNfQyJStp 2024-03-29T11:30:44.431 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.pcNfQyJStp 2024-03-29T11:30:44.520 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.pcNfQyJStp is 127KB 2024-03-29T11:30:44.572 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.pcNfQyJStp 2024-03-29T11:30:44.587 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689/data/mon.c.tgz 2024-03-29T11:30:44.587 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-03-29T11:30:44.642 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.7PC3uVZ9Sd 2024-03-29T11:30:44.642 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.7PC3uVZ9Sd 2024-03-29T11:30:44.735 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.7PC3uVZ9Sd 2024-03-29T11:30:44.821 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.7PC3uVZ9Sd is 127KB 2024-03-29T11:30:44.872 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.7PC3uVZ9Sd 2024-03-29T11:30:44.887 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689/data/mon.b.tgz 2024-03-29T11:30:44.887 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-03-29T11:30:44.906 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.8JIejr9IMC 2024-03-29T11:30:44.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.8JIejr9IMC 2024-03-29T11:30:45.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.8JIejr9IMC 2024-03-29T11:30:45.107 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.8JIejr9IMC is 127KB 2024-03-29T11:30:45.158 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.8JIejr9IMC 2024-03-29T11:30:45.174 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:30:45.174 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:45.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:45.257 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:45.297 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:45.297 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:30:45.299 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689/remote/smithi059/crash 2024-03-29T11:30:45.302 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:45.330 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689/remote/smithi172/crash 2024-03-29T11:30:45.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:45.364 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:30:45.364 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:45.374 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:45.441 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39890.log 2024-03-29T11:30:45.441 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:30:45.441 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38991.log 2024-03-29T11:30:45.441 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40127.log 2024-03-29T11:30:45.441 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39890.log.gz 2024-03-29T11:30:45.441 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38601.log 2024-03-29T11:30:45.442 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40205.log 2024-03-29T11:30:45.442 INFO:teuthology.orchestra.run.smithi059.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38991.log.gz 2024-03-29T11:30:45.442 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39929.log 2024-03-29T11:30:45.442 INFO:teuthology.orchestra.run.smithi059.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40127.log.gz 2024-03-29T11:30:45.442 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40205.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38601.log.gz 2024-03-29T11:30:45.442 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38250.log 2024-03-29T11:30:45.442 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40751.log 2024-03-29T11:30:45.442 INFO:teuthology.orchestra.run.smithi059.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40205.log.gz 2024-03-29T11:30:45.442 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38250.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39929.log.gz 2024-03-29T11:30:45.442 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38796.log 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40751.log: gzip 85.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.38250.log.gz -- 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.38640.log 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41336.log 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40751.log.gz 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38640.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38796.log.gz 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39616.log 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.41336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40400.log 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38640.log.gz 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39265.log 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41336.log.gz 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40400.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39616.log.gz 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40283.log 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39343.log 2024-03-29T11:30:45.443 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40400.log.gz 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38874.log 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39265.log.gz 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39343.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40283.log.gz 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39108.log 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40478.log 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39343.log.gz 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38523.log 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38874.log.gz 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40478.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39108.log.gz 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37974.log 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38169.log 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40478.log.gz 2024-03-29T11:30:45.444 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38523.log.gz 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38169.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37974.log.gz -5 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40085.log 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38445.log 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38169.log.gz 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38367.log 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38445.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40085.log.gz 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38091.log 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38367.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38445.log.gz 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39655.log 2024-03-29T11:30:45.445 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38367.log.gz 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- 86.1% /var/log/ceph/ceph-client.admin.40829.log 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38091.log.gz 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38913.log 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40829.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28535.log 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39655.log.gz 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40946.log 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38913.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40829.log.gz 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.tmp-client.admin.28535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38130.log 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28535.log.gz 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38913.log.gz 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41141.log 2024-03-29T11:30:45.446 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40439.log 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40946.log.gz 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.41141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39694.log 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38130.log.gz 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr: 86.4%/var/log/ceph/ceph-client.admin.40439.log: -- replaced with /var/log/ceph/ceph-client.admin.41141.log.gz 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39069.log 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39694.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40439.log.gz 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39851.log 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39694.log.gz 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.1.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39069.log.gz 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38052.log 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41297.log 2024-03-29T11:30:45.447 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38052.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39851.log.gz 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41180.log 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40985.log 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.41297.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38052.log.gz 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.41180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38757.log 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41297.log.gz 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41180.log.gz 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40361.log 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40985.log: /var/log/ceph/ceph-client.admin.38757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38289.log 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40985.log.gz 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38757.log.gz 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2024-03-29T11:30:45.448 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39733.log 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38289.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40361.log.gz 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41258.log 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38289.log.gz 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39733.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39538.log 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.41258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39421.log 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39733.log.gz 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39186.log 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41258.log.gz 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39421.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39538.log.gz 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38718.log 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34748.log 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39421.log.gz 2024-03-29T11:30:45.449 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39186.log: /var/log/ceph/ceph-client.admin.38718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39499.log 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34748.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39186.log.gz 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.38718.log.gz 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40868.log 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34748.log.gz 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38013.log 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38835.log 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39499.log.gz 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40517.log 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40868.log.gz 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39968.log 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38013.log.gz 2024-03-29T11:30:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40517.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38835.log.gz 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38952.log 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40517.log.gz 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39147.log 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39968.log.gz 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40322.log 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41063.log 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38952.log.gz 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39460.log 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39147.log.gz 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.41063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39382.log 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40322.log.gz 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40673.log 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41063.log.gz 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39382.log: gzip -5 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39460.log.gz 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40673.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39382.log.gz 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40673.log.gz 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40007.log 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39030.log 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41102.log 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39030.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40007.log.gz 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38209.log 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.41102.log: gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39030.log.gz 2024-03-29T11:30:45.452 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38562.log 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38209.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41102.log.gz 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38209.log.gz 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38562.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40634.log: gzip -5 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38562.log.gz 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39225.log 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.0.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41416.log 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40244.log 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr: 85.5%/var/log/ceph/ceph-client.admin.41416.log: -- replaced with /var/log/ceph/ceph-client.admin.39225.log.gz 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34655.log 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40166.log 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41416.log.gz 2024-03-29T11:30:45.453 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40244.log: /var/log/ceph/ceph-client.admin.34655.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40244.log.gz 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34655.log.gz 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41377.log 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.log: 85.9%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40166.log.gz 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.41377.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41377.log.gz 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39577.log 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38484.log 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.39577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40595.log 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38484.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39577.log.gz 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38484.log.gz 2024-03-29T11:30:45.454 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40046.log 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41219.log 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38679.log 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40595.log.gz 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.41219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39812.log 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40046.log.gz 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38679.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41219.log.gz 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38679.log.gz 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39772.log 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34354.log 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38406.log 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39812.log.gz 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34354.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41024.log 2024-03-29T11:30:45.455 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39772.log.gz 2024-03-29T11:30:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38406.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.34354.log.gz 2024-03-29T11:30:45.456 INFO:teuthology.orchestra.run.smithi059.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38406.log.gz 2024-03-29T11:30:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40907.log 2024-03-29T11:30:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.41024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38328.log 2024-03-29T11:30:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39304.log 2024-03-29T11:30:45.456 INFO:teuthology.orchestra.run.smithi059.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.41024.log.gz 2024-03-29T11:30:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38328.log: gzip -5 --verbose 85.5% -- /var/log/ceph/ceph-mon.c.log -- replaced with /var/log/ceph/ceph-client.admin.40907.log.gz 2024-03-29T11:30:45.456 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:30:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39304.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.38328.log.gz 2024-03-29T11:30:45.456 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39304.log.gz 2024-03-29T11:30:45.461 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:30:45.461 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34476.log 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34716.log 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34476.log: /var/log/ceph/ceph-client.admin.34716.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34476.log.gz 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34596.log 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34716.log.gz 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34756.log 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34676.log 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34756.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34596.log.gz 2024-03-29T11:30:45.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34676.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34756.log.gz 2024-03-29T11:30:45.463 INFO:teuthology.orchestra.run.smithi172.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34636.log 2024-03-29T11:30:45.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34676.log.gz 2024-03-29T11:30:45.463 INFO:teuthology.orchestra.run.smithi172.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34556.log 2024-03-29T11:30:45.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34636.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:45.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34556.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34636.log.gz 2024-03-29T11:30:45.463 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:30:45.463 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34516.log 2024-03-29T11:30:45.463 INFO:teuthology.orchestra.run.smithi172.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34556.log.gz 2024-03-29T11:30:45.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:45.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.34516.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34516.log.gz 2024-03-29T11:30:45.510 INFO:teuthology.orchestra.run.smithi172.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:30:56.740 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:30:56.924 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:30:56.932 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:31:00.765 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:31:03.560 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:31:03.573 INFO:teuthology.orchestra.run.smithi059.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:31:03.587 INFO:teuthology.orchestra.run.smithi059.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:31:03.599 INFO:teuthology.orchestra.run.smithi059.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:31:32.431 INFO:teuthology.orchestra.run.smithi059.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:31:34.075 INFO:teuthology.orchestra.run.smithi059.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:31:34.077 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-29T11:31:34.077 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m48.689s 2024-03-29T11:31:34.077 INFO:teuthology.orchestra.run.smithi059.stderr:user 2m42.521s 2024-03-29T11:31:34.077 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m5.539s 2024-03-29T11:31:56.748 INFO:teuthology.orchestra.run.smithi172.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:31:56.750 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-29T11:31:56.750 INFO:teuthology.orchestra.run.smithi172.stderr:real 1m11.324s 2024-03-29T11:31:56.750 INFO:teuthology.orchestra.run.smithi172.stderr:user 1m56.482s 2024-03-29T11:31:56.750 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m4.280s 2024-03-29T11:31:56.751 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:31:56.751 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689/remote/smithi059/log 2024-03-29T11:31:56.752 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:20.683 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689/remote/smithi172/log 2024-03-29T11:32:20.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:39.570 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:39.775 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4124cf57d070461094cdd732b7665626 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:39.778 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:32:39.789 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:32:39.821 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:32:39.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:39.837 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:39.915 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:32:39.925 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:32:39.926 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:39.928 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:39.942 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:39.944 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:39.946 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:39.946 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-03-29T11:32:39.947 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 +468us[ +459us] +/- 92ms 2024-03-29T11:32:39.947 INFO:teuthology.orchestra.run.smithi059.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 -103us[ -112us] +/- 55ms 2024-03-29T11:32:39.947 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -266us[ -266us] +/- 86ms 2024-03-29T11:32:39.947 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:39.948 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:39.949 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-03-29T11:32:39.949 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +463us[ +460us] +/- 92ms 2024-03-29T11:32:39.949 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -69us[ -72us] +/- 55ms 2024-03-29T11:32:39.949 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 -253us[ -255us] +/- 86ms 2024-03-29T11:32:39.949 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:39.950 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:32:39.990 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:32:39.990 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:32:40.024 DEBUG:teuthology.orchestra.run.smithi059:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:40.096 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:40.167 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:40.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:40.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:40.232 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:32:40.245 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:32:40.282 INFO:teuthology.task.internal:Duration was 1256.795768 seconds 2024-03-29T11:32:40.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:32:40.318 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:32:40.318 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:40.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:40.364 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:40.369 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:40.498 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:32:40.499 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2024-03-29T11:32:40.499 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:40.518 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-03-29T11:32:40.518 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:40.546 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:32:40.547 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:40.560 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:40.617 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:32:40.617 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:40.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:40.755 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:32:40.766 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:32:40.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:40.793 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:40.820 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:32:40.832 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:40.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:40.858 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2024-03-29T11:32:40.886 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-03-29T11:32:40.912 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:40.948 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:40.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:40.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:32:40.976 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:32:40.977 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689/remote/smithi059 2024-03-29T11:32:40.978 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:41.023 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689/remote/smithi172 2024-03-29T11:32:41.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:41.058 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:32:41.058 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:41.061 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:41.107 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:32:41.122 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:32:41.122 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:32:41.155 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:32:41.156 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:41.158 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:41.170 INFO:teuthology.orchestra.run.smithi059.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:41.173 INFO:teuthology.orchestra.run.smithi172.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:41.174 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:32:41.190 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:32:41.256 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:32:41.284 DEBUG:teuthology.nuke:shortname: smithi059 2024-03-29T11:32:41.285 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:41.311 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:07.175750', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpM4AhB3Xc7/K7D8W6puQ5pHgaOxyi5wEP+yhHj9ifzq6krw/XMR2L09TlXLiV16kErqxR0s9oT5hpN8e/DsHQ='} 2024-03-29T11:32:41.315 DEBUG:teuthology.nuke:shortname: smithi172 2024-03-29T11:32:41.315 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:41.337 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622689', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:07.177226', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLKHThZ4r0Bb1ok8Y8VwtGYVX4plSSApvkvHIR3vWr7lFP6SSIFvSZm56wznkB3V50jsyn0stxgaz0jPKH27bzk='} 2024-03-29T11:32:41.363 INFO:teuthology.orchestra.console.smithi059:Power off 2024-03-29T11:32:41.363 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:41.386 DEBUG:teuthology.orchestra.console.smithi059:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:41.387 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:41.406 INFO:teuthology.orchestra.console.smithi172:Power off 2024-03-29T11:32:41.407 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:41.418 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-29T11:32:41.431 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:41.432 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:41.456 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-29T11:32:45.419 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:45.457 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:45.531 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-29T11:32:45.567 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-29T11:32:49.534 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:49.567 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:49.646 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-29T11:32:49.681 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-29T11:32:53.650 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:53.682 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:53.761 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-29T11:32:53.795 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2024-03-29T11:32:53.862 INFO:teuthology.orchestra.console.smithi059:Power off completed 2024-03-29T11:32:53.896 INFO:teuthology.orchestra.console.smithi172:Power off completed 2024-03-29T11:32:54.069 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1256.7957680225372 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4124cf57d070461094cdd732b7665626 status: fail success: false 2024-03-29T11:32:54.070 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:32:54.156 INFO:teuthology.run:FAIL